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To better monitor children and families served by state child 
welfare agencies, Congress authorized matching funds for the development of 
statewide automatic child welfare information systems (SACWIS) and required 
that U.S. Department of Health and Human Services (HHS) compile information 
on children served by state agencies. This report to Congress reviews states ' 
experiences in developing SACWIS and HHS * s role in assisting in their 
development; factors affecting reliability of data that states collect and 
report on children served by child welfare agencies, and HHS * s role in 
ensuring reliability; and practices that child welfare agencies use to 
overcome challenges associated with SACWIS development and data reliability. 
Information was collected through a Web-based survey of all 50 states and the 
District of Columbia, through state visits (Colorado, Iowa, New York, North 
Carolina, Oklahoma), through a review of SACWIS and Adoption and Foster Care 
Analysis and Reporting System reports, and through interviews of HHS, state, 
and local officials. Findings indicated that 47 states are developing or 
operating a SACWIS, but many continue to face challenges developing their 
systems. Although most state officials recognized benefits of developing 
SACWIS, states reported a median delay of 2.5 years beyond the timeframes set 
for completion. Difficulties cited included challenges receiving state 
funding and creating systems reflecting work processes. Several factors 
affected states' ability to collect and report reliable adoption, foster 
care, and child abuse and neglect data, including insufficient caseworker 
training and inaccurate and incomplete data entry. Despite HHS assistance, 
many states report ongoing challenges, such as the lack of clear and 
documented guidance on how to report child welfare data. Some states reported 
using a variety of practices to address the challenges associated with 
developing SACWIS and improving data reliability. The report recommends that 
HHS consider ways to enhance the guidance and assistance offered to states. 
The report’s four appendices include information on the study methodology, 
stages of SACWIS development, and HHS comments. (Contains a 10-item 
bibliography and a list of related Government Accounting Office [GAO] 
publications.) (KB) 
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Highlights of GAO-03-809, a report to 
Congressional Requesters 



Most States Are Developing Statewide 
information Systems, but the Reiiability 
of Child Welfare Data Could Be Improved 



Why GAO Did This Study 

To better monitor children and 
families served by state child 
welfare agencies, Congress 
authorized matching funds for the 
development of statewide 
automated child welfare 
information systems (SACWIS) and 
required that the Department of 
Health and Human Services (HHS) 
compile information on the 
children served by state agencies. 
This report reviews (1) states’ 
experiences in developing child 
welfare information systems and 
HHS’s role in assisting in then- 
development, (2) factors that affect 
the reliability of data that states 
collect and report on children 
served by their child welfare 
agencies and HHS’s role in 
ensuring the reliability of those 
data, and (3) practices that child 
weltoe agencies use to overcome 
challenges associated with SACWIS 
development and data reliability. 



What GAO Recommends 



In order to improve the reliabiUty 
of state-reported child welfare 
data, GAO recommends that the 
Secretary of HHS consider ways to 
enhance the guidance and 
assistance offered to states to help 
them overcome the key challenges 
in collecting and reporting child 
welfare data In commenting on 
this report, HHS generally agreed 
with GAO’s findings and 
commented that the report 
provides a useful perspective of the 
problems states face in collecting 
data and of HHS’s effort to provide 
ongoing technical assistance to 
improve child welfare data. 

www.gao.gov/cgi-bin/getrpt7GAO-03-809. 

To view the full product, including the scope 
and methodology, click on the link above. 

O tore information, contact Cornelia M. 

F R I r y at (202) 512-8403 or 
— yC@gao.gov. 



What GAO Found 

HHS reported that 47 states are developing or operating a SACWIS, but many 
continue to face challenges developing their systems. Most state officials 
said they recognize the benefit their state will achieve by developing 
SACWIS, such as contributing to the timeliness of child abuse and neglect 
investigations; however, despite the availabilily of federal funds since 1994, 
states reported a median delay of 2-‘/2 years beyond the timeframes they set 
for completion. States reported that they encountered some difficulties 
during SACWIS development, such as challenges receiving state funding and 
creating a system that reflected their work processes. In response to some of 
these challenges, HHS has provided technical assistance to help states 
develop their systems and conducted on-site reviews of SACWIS to verify 
that the systems meet federal requirements. 

Despite efforts to implement comprehensive information systems, several 
factors affect the states’ ability to collect and report reliable adoption, foster 
care, and child abuse and neglect data. States responding to GAO’s survey 
and officials in the 5 states GAO visited reported that insufficient 
caseworker training and inaccurate and incomplete data entry affect the 
quality of the data reported to HHS. In addition, states reported technical 
challenges reporting data. Despite HHS’s assistance, many states report 
ongoing challenges, such as the lack of clear and documented guidance on 
how to report child welfare data. In addition, although states were mandated 
to begin reporting data to the Adoption and Foster Care Analysis and 
Reporting System (AFCARS) in 1995, few reviews of states’ AFCARS 
reporting capabilities have been conducted to assist states in resolving some 
of their reporting challenges. 

Some states are using a variety of practices to address the challenges 
associated with developing SACWIS and improving data reUabiUty. For 
example, 44 states included caseworkers and other system users in the 
design and testing of SACWIS, and 28 states reported using approaches to 
help caseworkers identify and better understand the data elements that are 
required for federal reporting. 
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United States General Accounting Office 
Washington, DC 20548 



July 31, 2003 

The Honorable Charles E. Grassley 
Chairman, Committee on Finance 
United States Senate 

The Honorable Tom DeLay 

Majority Leader 

House of Representatives 

Recent news reports of tragedies involving child welfare agencies across 
the country highlight the long-standing problems states have had 
monitoring children in their care. Given that hundreds of thousands of 
children are found to be victims of abuse and neglect and are estimated to 
spend some time in foster care each year, the Congress required that the 
Department of Health and Human Services (HHS) compile information on 
the children served by state agencies and authorized federal funds to 
match those of states for use in the development of state child welfare 
information systems. Since 1994, designated federal matching funds have 
been available to states to develop and implement comprehensive case 
management systems — Statewide Automated Child Welfare Information 
Systems (SACWIS)— to manage their child welfare cases as well as to 
report child abuse and neglect, foster care, and adoption information to 
the federal government. States have the option to implement a SACWIS or 
develop different information systems without using SACWIS funds to 
support their child welfare agencies and collect information on their child 
welfare cases. Regardless of the type of system a state develops, child 
welfare caseworkers at the county or local level are the key personnel 
who collect and document information on children and families served by 
child welfare agencies, in addition to performing a wide range of services 
to protect children— such as investigating child abuse or neglect reports or 
providing support services to maintain the children in their homes. 
Currently, HHS compiles state-reported child welfare data in two 
databases: the Adoption and Foster Care Analysis and Reporting System 
(AFCARS) and the National Child Abuse and Neglect Data System 
(NCANDS). HHS relies on the information available in its databases to 
analyze and track children’s experiences in the child welfare system, to 
determine states’ performance on federal child welfare outcome measures, 
and to report to Congress on children’s well being and child welfare 
experiences. 
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Results in Brief 



ERIC 



Because of your interest in how states have developed systems to collect 
and report information on the children they serve and the reliability of that 
information, you asked us to determine (1) states’ experiences in 
developing child welfare information systems and HHS’s role in assisting 
in their development; (2) factors that affect the reliability of data that 
states collect and report on children served by their child welfare 
agencies, and HHS’s role in ensuring the reliability of those data; and 
(3) practices that child welfare agencies use to overcome challenges 
associated with SACWIS development and data reliability. 

To conduct our work, we surveyed all 50 states and the District of 
Columbia regarding their experiences in developing and using information 
systems and their ability to report data to HHS. We received responses 
from 49 states and the District of Columbia,* although all states did not 
respond to every question. We also reviewed a variety of HHS documents, 
including the protocol and reports for its reviews of SACWIS systems and 
states’ AFCARS reporting capabilities. In addition, we visited 5 states — 
Colorado, Iowa, New York, North Carolina, and Oklahoma — to obtain first- 
hand information on their experiences developing SACWIS and reporting 
data to HHS. We selected these states to represent geographic diversity 
and different stages of SACWIS implementation. Finally, we interviewed 
HHS officials and child welfare and data experts and reviewed relevant 
literature. We conducted our work between June 2002 and June 2003 in 
accordance with generally accepted government auditing standards. A 
more detailed discussion of our scope and methodology appears in 
appendix 1. 



HHS reported that 47 states are developing or operating a SACWIS, but 
many continue to face challenges despite HHS’s oversight and technical 
assistance. Most states are using federal SACWIS funds and are in various 
stages of development. States reported in our survey that they have spent 
approximately $2.4 billion in federal, state, and local funding on SACWIS. 
Most state officials said they recognize the benefit their state will achieve 
by developing SACWIS, such as contributing to the timeliness of child 
abuse and neglect investigations; however, despite the availability of 
federal funds since 1994, many child welfare agencies lag behind the 



‘Throughout this report, references to state survey responses include the District of 
Columbia. Forty-six of these states reported that they are developing or operating a 

SACWIS. Nevada, which HHS reported has an operational SACWIS, did not respond to our 
survey. 
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timeframes they set for completion, with delays ranging from 2 months to 
8 years. Forty-two of the 46 states responding to our survey that they are 
developing SACWIS reported at least some challenge obtaining state 
funding. In Iowa, for example, state officials reported that insufficient 
state funds delayed them in making the necessary modifications to meet 
federal requirements for system completion. Some states had difficulties 
developing a system that met the state child welfare agency’s needs 
statewide. For example, state officials in New York— a state where the 
counties are responsible for administering child welfare services said 
that building a uniform system was stalled when significant frustration 
with the system’s design led some county officials to request that the state 
stop SACWIS development. In addition, 32 states reported at least some 
challenge securing information technology contractors with knowledge of 
child welfare practice to develop their SACWIS. In response to some of 
these challenges, HHS has provided technical assistance to help states 
develop their systems and conducted on-site reviews of SACWIS to verify 
that the systems meet all federal requirements. For example, at the time of 
our review, HHS had conducted on-site reviews in 26 states with 
operational SACWIS to ensure that the systems met all federal 
requirements and to offer assistance to states that faced challenges 
completing the development of their SACWIS. 

Several factors affect states’ ability to collect and report reliable data on 
children served by state child welfare agencies, and some problems exist, 
such as a lack of clear and documented guidance, with HHS’s oversight 
and technical assistance. Almost all of the states responding to our survey 
reported that insufficient caseworker training and inaccurate and 
incomplete data entry into their information system affect the quality of 
the data reported to HHS. Although most states reported these as separate 
factors, HHS and the states we visited found that insufficient training and 
inaccurate and incomplete data entry are often linked. In addition, 36 of 
the 50 states that responded to our survey reported that technical 
challenges, such as matching their state data element definitions to HHS’s 
data categories, affected the quality of the data that they report to the 
federal government. Similarly, during assessments of 6 states’ compliance 
with AFCARS reporting standards, HHS found that these issues affect data 
reliability. Despite HHS’s assistance in helping states improve their data, 
such as testing state data quality and providing the results to the states to 
aid them in resubmitting data, states report ongoing challenges. For 
example, 41 of the 50 states responding to our survey reported that a lack 
of clear and documented guidance from HHS affects the quality of the data 
they report to AFCARS, and 25 states said the lack of clear, documented 
guidance also affected the data reported to NCANDS. In addition, although 
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states were mandated to begin reporting AFCARS in 1995, few reviews of 
states AFCARS reporting capabilities have been conducted to assist states 
in resolving some of their challenges. 

Some states are using a variety of practices to address the challenges 
associated with developing SACWIS and improving data reliability, 
although no formal evaluations are available on their effectiveness. To 
overcome the challenge of developing a system to meet statewide needs, 
many states relied on caseworkers and other system users for input on 
design and testing of SACWIS. Few states reported that they devised 
strategies to overcome the other challenges, such as limited funding and 
finding contractors with knowledge of child welfare. However, Oklahoma 
child welfare officials — in order to maximize the limited state funding for 
maintaining their SACWIS — reported saving $1 million each year by hiring 
some of the contractors who developed their SACWIS as permanent staff. 
To improve data reliability, the 5 states we visited routinely review their 
data to identify data entry errors so that managers can ensure that the 
missing data are entered appropriately. In addition, some states reported 
that frequent use of the data, such as publishing periodic management 
reports detailing local offices’ performance on outcome measures, helps 
caseworkers understand the importance of entering timely information. 

In order to improve the reliability of state-reported child welfare data, we 
are recommending that the Secretary of HHS consider ways to enhance 
the guidance and assistance offered to states to help them overcome the 
key challenges in collecting and reporting child welfare data. In 
commenting on this report, HHS’s Administration for Children and 
Families (ACF) generally agreed with our findings and commented that 
the report provides a useful perspective of the problems states face in 
collecting data and of ACF’s effort to provide ongoing technical assistance 
to improve the quality of child welfare data. In response to our 
recommendation, ACF said that we did not recognize the long-term efforts 
it has taken to provide AFCARS and NCANDS guidance. ACF also noted 
that the data definitions need to be updated and revised and said it is 
currently in the process of revising the AFCARS regulations to further 
standardize the information states are to report — which we acknowledge 
in our report. Further, ACF added that although staff turnover in state 
child welfare agencies is a significant contributor to data quality issues, we 
did not focus on this as a significant factor. However, because we recently 
issued a detailed report on a variety of caseworker issues, we primarily 
focused in this report on the key data entry challenges caseworkers face 
and refer readers to our previous work for additional information on 
challenges related to caseworker recruitment and retention and their 
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affect on child welfare agencies. ACF commented that it is firmly 
committed to continue to support the states and to provide technical 
assistance and other guidance as its resources will permit. We believe that 
the recent activities to formally obtain, document, and incorporate 
feedback from the states with regard to collection and reporting adoption 
and foster care data represent are a step towards improving states data. 
Our recommendation encourages HHS to consider ways to enhance the 
guidance and assistance already offered to states as a step to helping them 
better comply with the reporting requirements. 



Background 



ACF’s Children’s Bureau is responsible for the administration and 
oversight of federal funding to states for child welfare services under 
Titles IV-B and IV-E of the Social Security Act. However, the monitoring of 
children served by state child welfare agencies is the responsibility of the 
state agencies that provide the services to these children and their 
families. Child welfare caseworkers at the county or local level are the key 
personnel responsible for documenting the wide range of services offered 
to children and families, such as investigations of abuse and neglect; 
treatment services offered to families to keep them intact and prevent the 
need for foster care; and arrangements made for permanent or adoptive 
placements when children must be removed from their homes. 
Caseworkers are supported by supervisors who typically assign new cases 
to workers and monitor caseworkers’ progress in achieving desired 
outcomes, analyzing and addressing problems, and making decisions 
about cases. 

A number of efforts at the national level have been taken to implement 
comprehensive data systems that capture, report, and analyze the child 
welfare information collected by the states (see table 1 for information on 
national data systems as well as information on state systems). 
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System 


History 


Use 


oiaie information systems 


btatewiae Automated Child Welfare Information 
System (SACWIS) 


The Omnibus Budget Reconciliation Act 
(OBRA) of 1993 authorized the use of an 
enhanced federal financial participation 
(FFP) rate of 75 percent to assist states 
develop uniform automated Information 
systems — SACWIS — that support the 
administration of services offered under their 
child welfare programs." The enhanced FFP 
was available Initially from federal fiscal 
years 1994 through 1996 and subsequently 
extended through federal fiscal year 1997. 

, After 1997, states receive a 50 percent 
match for SACWIS-related activities. 

Funding approval for SACWIS Is based on 
states’ estimated costs for development and 
operation, and no time limits are placed on 
the receipt of federal funding. 


A SACWIS Is designed and 
developed for use by states’ 
caseworkers and other 
personnel for the purposes of 
establishing an electronic case 
file for children and families 
served by the state child welfare 
agency. Some of the data 
captured In SACWIS are 
reported to HHS. 


iMaiionai data systems 


voluntary cooperative Information System (VCIS) 


With funding from HHS, the American Public 
Welfare Association — now known as the 
American Public Human Services 
Association— established VCIS in an effort 
to compile state-specific child welfare data. 
States voluntarily reported aggregate-level 
data on the characteristics of children in 
foster care and those adopted from state 
child welfare agencies. VCIS compiled data 
between 1982 and 1994. 


The data available in VCIS was 
used to produce reports at the 
national level on the 
characteristics of children in 
foster care and those adopted 
from state child welfare 
agencies. However, the data 
reported to VCIS were 
Inconsistent.^" Following the 
Implementation of AFCARS, 
states no longer reported to 
VCIS. 
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History 



System 

Adoption and Foster Care Analysis and Reporting 
System (AFCARS) 



National Child Abuse and Neglect Data System 
(NCANDS) 



In 1986, Congress added Section 479 to 
Title IV-E of the Social Security Act, which 
required HHS to establish and implement a 
mandatory foster care and adoption data 
collection system. Despite the legislative 
requirement that HHS complete final 
regulations for the system by the end of 
calendar year 1988, regulations were not 
issued until December 1993. States began 
reporting to AFCARS in 1995 on the 
demographic characteristics of adoptive and 
foster children and their parents as well as 
foster children’s type of placement and 
permanency goals. States are required to 
report 66 foster care and 37 adoption data 
elements for each child in care during 
6-month periods. Between 1998 and 
2001, states not meeting certain reporting 
standards were subject to penalties. The 
penalties were rescinded in January 2002 
followinq an appeal filed by 1 2 states. 



In 1988, Congress amended the Child 
Abuse Prevention and Treatment Act 
(CAPTA) by directing HHS to establish a 
national data collection and analysis system 
consisting of state child abuse and neglect 
information. Walter R. McDonald & 
Associates was awarded the contract to 
compile and analyze the state-reported data. 
States began voluntarily reporting annually 
in 1990; however, the 1996 CAPTA 
amendments directed states to report 
certain data to receive CAPTA grant 
funding. States submit either child-specific 
records or aggregate-level data. In 2001 — 
the most recently available data from HHS — 
39 states and the District of Columbia 
submitted child-specific data, which includes 
the demographics of the children and their 
perpetrators, the types of maltreatment, 
investigation or assessment findings, risk 
factors, and services provided as a result of 
the investigation or assessment. The 
remaining 1 1 states submitted aggregate 
data. 



Use 

Some of the key uses of 
AFCARS data include: 



Calculating the national standard 
for five federal outcome 
measures — incidence of child 
abuse and/or neglect in foster 
care, foster care re-entries, 
stability of foster care 
placements, length of time to 
achieve reunification, and length 
of time to achieve adoption. 



The publication of the annual 
Child Welfare Outcomes Report. 



Determining the allocation of 
funds in the adoption incentive 
program and Chafee Foster Care 
Independence Program.*" 

Some of the key uses of 
NCANDS data include: 



Calculating the national standard 
for two federal outcome 
measures on child safety — 
recurrence of maltreatment and 
incidence of child abuse and/or 
neglect in foster care. 



The publication of the annual 
Child Maltreatment Report. 



The publication of the annual 
Child Welfare Outcomes Report. 
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National Youth in Transition Database (NYTD) 



nidiury 



Use 



In response to requirements included in the 
Foster Care Independence Act of 1999, 

HHS has proposed to develop the NYTD to 
capture more detailed data than is currently 
reported to AFCARS on older foster children 
who receive independent living services. 
Such services include life-skills training in 
financial management or career planning. 
HHS reported to Congress in September 
2001 that nationwide implementation of 
NYTD was planned for October 2003, but 
recent HHS information indicates that the 
actual implementation date will likely be 
2005 or 2006. Once implemented, states will 
be required to report to NYTD. 



In response to the legislation, 
data available in the NYTD will 
be used, in part, to track (1) the 
number and characteristics of 
older youth receiving 
independent living services, 

(2) the type and quantity of 
services states provided, and 

(3) state performance on the 
outcome measures required by 
the legislation. 



SalE^f'a percent FFP included planning, design, development, and 

stallation of a SACWIS. Other costs, such as operation of the SACWIS, were covered at 50 percent. 

“See www.acf.hhs.gov/programs/cb/dis/vcis/maintoc.htm tor more complete details on the 

shortcomings of the data available in VCIS. i-ump,e,e oeiaiis on the 

'The total number of finalized adoptions reported to AFCARS are used to determine the amnimt if 
any, of the adoption incentive payments awarded to states. In addition, AFCARS data are used to 
determine the ratio of the number of children in foster care in each state to the total Jumbe^^^^ 

mlTende“ Foster Care°' 



To qualify for federal funding for SACWIS, states must prepare and submit 
an advance planning document (APD) to ACPs Children’s Bureau, in 
which they descnbe the state’s plan for managing the design, development 
implementation, and operation of a SACWIS that meets federal 
requirements and state needs in an efficient, comprehensive, and cost- 
effective manner.^ In addition, the state must establish SACWIS and 
progr^ performance goals in terms of projected costs and benefits in the 
States are required to submit separate APDs for the planning and 
development phases, in addition to periodic updates. 



When states choose to develop information systems that include other human services 

enforcement, or Medicaid, states must submit APDs’to 
Technolo"ev^ agency. In a heanng held last year before the Subcommittee on 

Policy, House Committee on Government Reform, we 
„ , , ^ ederal agencies do not have systems to monitor states requests for 

federal approval and funding through the life cycle of a state request. (See U S General 
Accounting Office, Human Semnces: Federal Approval and Fxmdinp Processes for Slates’ 
Infoima.tion System, GAO-02-347T (Washington, D.C: July 9, 2002.)) 
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Since the administration and structure of state child welfare agencies vary 
across the nation, states can design their SACWIS to meet their state 
needs, as long as states meet certain federal requirements. Federal funding 
is available to states for SACWIS that 

• meet the requirements for reporting AFCARS data to HHS; 

• to the extent practicable, are capable of linking with the state data 
collection system that collects information on child abuse and neglect; 

• to the extent practicable, are capable of linking with, and retrieving 
information from, the state data collection system that collects 
information on the eligibility of individuals under Title IV-A — Temporary 
Assistance for Needy Families; and 

• provides for more efficient, economical, and effective adnunistration of 
the programs carried out under a state’s plans approved under Titles IV-B 
and IV-E of the Social Security Act. 

A SACWIS must operate uniformly as a single system in each state and 
must encompass all entities that administer programs provided under 
Titles IV-B and IV-E. In some cases, HHS will allow the statewide system 
to link to another state system to perform required functions, such as 
linking to financial systems to issue and reconcile payments to child 
welfare service providers. The state’s APD must describe how its SACWIS 
will link to other systems to meet the requirements in the SACWIS 
regulations. 

In addition to monitoring the APDs of the states that are developing 
SACWIS, HHS reviews state information systems through formal SACWIS 
assessment reviews and the Child and Family Services Reviews (CFSR)— a 
federal review process to monitor states’ compliance with child welfare 
laws and federal outcome measures. The formal SACWIS reviews are 
conducted by ACFs Children’s Bureau to determine if a state has 
developed and implemented all components detailed in the state’s APD 
and if the system adheres to federal requirements. The CFSR assesses 
statewide information systems, along with other systemic factors, to 
determine if the state is operating a system that can readily identify the 
status, demographic characteristics, location, and goals for placement of 
every child who is in foster care. This systemic factor is reviewed in all 
states, regardless of whether the state is developing a SACWIS or the stage 
of system development. According to results from the fiscal years 2001 and 
2002 CFSRs, 4 of the 32 states in which HHS reviewed were not in 
substantial conformity on the statewide information system indicator. 
These 4 states must address how they will come into conformity with this 
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factor in a program improvement plan, 
reviews in 2 of these states. 



HHS has also conducted SACWIS 



Most States Are 
Developing SACWIS, 
but Challenges 
Remain Despite HHS’s 
Oversight and 
Technical Assistance 



While 47 states are developing or operating a SACWIS, many challenges 
remain despite HHS’s oversight and technical assistance. Since 1994, 
states reported that they have spent approximately $2.4 billion in federal 
state, and local funding on SACWIS. While most state officials we 
interviewed and those responding to our survey said that they recognize 
the benefits their state will achieve by developing a statewide system, 
many states reported that the development of their SACWIS is delayed 
between 2 months and 8 years beyond the time frames the states set for 
completion, with a median delay of 2->/2 years. Most states responding to 
our survey faced challenges, such as obtaining state funding and 
developing a system that met the child welfare agency’s needs statewide 
In response to some of these challenges, HHS has provided technical 
assistance to help states develop their systems and conducted on-site 
SACWIS reviews to verify that the systems meet all federal requirements 



States Are Using Federal 
and State Funds and 
Various Participants to 
Develop MultiComponent 
SACWIS 



Currently, 47 states are developing or operating a SACWIS and are in 
various stages of development — ranging from planning to complete. The 
states responding to our survey reported using approximately $13 billion 
m federal funds^ and approximately $1.1 bUlion in state and local funds^ for 

HHS estimated that it allocated approximately 
$821 mUion between fiscal years 1994 and 2001 in SACWIS developmental 
funds and $173 million between fiscal years 1999 and 2001 in SACWIS 



Wy-four states provided information on the total amount of federal funds they received 
to develop and operate SACWIS. Alaska, Hawaii, Missouri, North Carolina Texas and 

S reoort H fi information. Nevada did not respond to our survey 

State-reported figures may include some funding allocated in fiscal year 2003 since the 
survey was issued in October 2002 and completed as late as December 2002. 

information on the total amount of state funds used to develop 
^d operate SACWIS. Arkansas, Hawaii, Missouri, North Carolina, Texas, and Vermont did 
not report state funding mformation. Nevada did not respond to our survey State-renorted 

ScToKo^^d"”"'" n 

m October 2002 and completed as late as December 2002. 

allocated by HHS to 49 states and the District of 
Columbia. Hawaii did not take any federal money for SACWIS development. 
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operational funds.® The total amount of federal funding provided to states 
for SACWIS is unknown because states claimed operational costs as a part 
of their Title IV-E administrative expenses prior to 1999.’ Although the 
federal government matched state funding at an enhanced rate of 
75 percent beginning in 1994, many states did not apply for federal funding 
or begin SACWIS development until 1996 and 1997 when more than 
$467 million — the bulk of federal funds — were allocated. Most states were 
still developing their SACWIS by the time enhanced funding expired in 
1997, after which states could receive a 50 percent FFP for SACWIS 
development and operation. Although 47 states are currently developing or 
operating a SACWIS, all states except Hawaii received some federal 
SACWIS funds. For example, according to figures provided by HHS, North 
Carolina received approximately $9.6 million in developmental funds and 
North Dakota received approximately $2.4 million in developmental funds 
and $245,000 in operational funds for SACWIS, but both states 
encountered difficulties that prevented them from completing their 
systems. In these situations, HHS entered into negotiations with the states 
about the amount of money that the states must return to the federal 
government. 

In order to track states^ SACWIS development, HHS places them in six 
categories that identify their stage of development (see table 2). States are 
required to submit APD updates periodically, which inform HHS of their 
progress in developing SACWIS. See appendix II for a complete list of 
states’ phases of development. Although most states continue to advance 
in the development of their systems, some encounter problems that cause 
HHS to recategorize them into a lower stage of development. In 
Pennsylvania, for example, the child welfare agency encountered 
difficulties, such as inadequate computer software to support a 
comprehensive SACWIS, after attempting to implement its SACWIS in 



®This figure includes operational funds allocated to 35 states. States begin claiming 
operational costs when some or all components of their SACWIS are operating in loc^ 
offices. Operational activities include routine maintenance, minor enhancements, and other 
changes that do not significantly increase or modify the fimctionality of the system. 

’According to HHS officials, prior to fiscal year 2000, states reported SACWIS operational 
expenses as part of their Title IV-E administrative expenses because the claims sheet states 
used for reporting did not have a separate column for SACWIS operational expenditures. In 
fiscal year 2000, states were required to use a claims sheet that was reformatted to provide 
space for SACWIS operational expenditures. In addition, an HHS official explained that the 
difference between the state-reported figures and the federal figures may be due to states 
claiming some SACWIS expenses under different programs, such as Title IV-E 
administrative funds, rather than separately as SACWIS expenses. 
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2000. Due to these problems, the state is in the process of shutting down 
the system and has resubmitted an APD for a new system to HHS for 
review and approval for further federal funding. According to figures 
provided by HHS, Pennsylvania has received approximately $9.7 million in 
federal funding thus far. In addition, while HHS may classify a state system 
as complete following an assessment of their SACWIS, a state may make 
additional changes to the system since SACWIS, like other computer 
systems, continually evolve as technology and child welfare practices 
change. States can claim federal funding for these changes as operational 
expenses. For example, Oklahoma’s SACWIS was the first system to be 
determined complete, but it has made enhancements to its system since 
HHS found the system in compliance with federal requirements in 1998. In 
addition, Oklahoma is currently considering moving to a Web-based 
system. An HHS official reported that such changes do not need prior 
approval unless they are in excess of $5 million. 



Table 2: Number of States in 


Various Stages of SACWIS Development 


Stage 


Number of states 


Complete® 


5 


Operational*' 


24 


Partially operational' 


'g 


Implementation*^ 


2 


Planning® 


7 


No SACWIS' 


4 


Source: HHS. “ 



Note: Status is as of May 22, 2003. 

'SACWIS assessment process is completed, and all functional requirements and specifications set 
forth in the APD are either included in the system or in an accepted corrective action plan. 

‘All functional requirements and specifications in APD are inciuded in system, and system is 
functionai statewide, but state has not completed SACWIS assessment or is working on other issues. 

statewide*'" system to field sites or still adding functions to systems that are operational 

"In active design and deveiopment, even if deiayed whiie waiting to resolve problems such as funding. 
"Working through options for a SACWIS. 

Have never pursued SACWIS funding or have abandoned plans to develop a system. 



In developing a system, states have considerable flexibility in the design of 
their SACWIS. According to HHS officials, a state should be using its 
SACWIS as a case management tool that uses automation to support the 
various aspects of state child welfare programs, such as recording child 
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protection, out of home care, and foster care and adoption services. To 
further assist child welfare practice, states have designed their systems to 
follow the natural flow of child welfare practice in their state and have 
added design features to help track key events dming a case. For example, 
in Iowa child welfare work is divided between child abuse and neglect 
investigations and ongoing case management for children brought into the 
care of the child welfare agency. As a result, Iowa designed a SACWIS to 
reflect this work process by linking two databases one to record child 
abuse and neglect information and one to record ongoing case records— 
that share information with one another.® In Rhode Island, the SACWIS 
was designed to alert caseworkers if an alleged perpetrator has been the 
subject of three reports of abuse or neglect within 1 year. Regardless of 
the findings of each report, this alert notifies the caseworker to initiate an 
investigation when a third report is received. 



Since many states are in different phases of SACWIS development, their 
systems currently support to varying degrees a variety of child welfare and 
administrative components (see table 3). According to HHS, while the 
components represented in table 3 are required for a state’s SACWIS to be 
considered compliant with federal guidance — either through an interface 
or built within the system — some of the subcomponents, such as a 
function that helps caseworkers manage their caseloads, are optional. 

HHS has encouraged states to automate as many functions as possible in 
the SACWIS in an effort to cut down on the additional paperwork or 
duplicative steps inherent in manual data collection. One of these services, 
tracking independent living, is becoming more important for states as HHS 
decides how to implement the Foster Care Independence Act of 1999 and 
considers the development of the NYTD.® Some states have already started 
collecting data on older youth and the services they receive. Currently, 

27 states reported in our survey that they are at some stage of using their 
SACWIS to track independent living services, and an additional 14 states 
plan to include this component in their system in preparation for the 



®Although the Iowa state officials’ described their SACWIS as including the chUd abuse and 
neglect system, HHS commented on a draft of this report that it does not view the child 
abuse and neglect system as part of the state’s SACWIS. However, HHS said that the state 
has met the SACWIS requirement in this area by building an interface between the two 
systems. 

®The Foster Care Independence Act of 1999 increased federal support to states for 
independent living programs — programs designed to assist youth who are identified as 
likely to remain in foster care until age 18. Independent living services can include 
education or training necessary for the youth to obtain employment. 
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requirements. However, 21 of the 46 states reporting to our survey that 
they are developing or operating a SACWIS reported that they would have 
to make substantial changes to their SACWIS in order to capture this 
information. 



^^blea^electedSACwIShHdWelfare^IIdASS 


Service 


Fully or partially 
operational in SACWIS 


Planned 
for SACWIS 


Child Welfare Services 


Child protection® 


38 


5 


Out of home care‘s 


35 


8 


Adoption 


34 


9 


Independent living 


27 


14 


Intensive home-based services' 


27 


13 



Workload management 


32 


8 


IV-E eligibility"^ 


29 


14 


Foster care maintenance 
payments 


28 


14 


Adoption assistance payments 


25 


17 


Contract provider payment 


24 


15 


Source: GAO survey. 



Note: Based on responses from 46 states developing or operating a SACWIS. The rows for the 
columns fully or partially operational’’ and "planned” do not add to 46 because the respondents mav 
have answered not supported,” “don’t know,” or “no answer.” ^ 

Child protection includes services such as intake and screening, investigation, and disposition. 

"Out of home care includes things such as foster care, group homes, and residential placement. 

'Intensive home-based services include efforts to avoid placing a child In foster care. 

'IV-E funding is available for foster care, adoption, and Independent living services. 



To assist with the design of their SACWIS, states relied on a number of 
different participants including internal users, such as caseworkers and 
managers, information technology (IT) staff, and contractors. Most states 
found these participants to be extremely or very helpful in the process 
(see table 4). In Oklahoma, for example, 150 child welfare staff from the 
field worked closely with the contractor in intensive work group sessions 
to design and test the system. To complement the caseworkers’ knowledge 
of child welfare practice, 43 states relied on IT staff. In Colorado, for 
example, IT staff said that during SACWIS design and development, they 
shared office space with program staff that had been assigned to help with 
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SACWIS development. This co-location of staff aided in the exchange of 
information pertaining to the development of the system. Finally, 42 states 
reported that they hired private contractors to conduct a large part of 
SACWIS design and development. The contractors helped states meet 
federal requirements, designed the system with state specific options, 
wrote the necessary software, tested and implemented the system, and 
trained users. 



Table 4: Commonly Used 
Helpfulness 


SACWIS Development Participants and Their Level of 


SACWIS development 
participant 


Number of states 
using participant 


Number of states rating participant 
as extremeiy or very helpful 


Internal users 


44 


41 


IT staff 


43 


31 


Private contractors 


42 


37 



Source: GAO survey. 



Note: Based on responses from 46 states developing or operating a SACWIS. 



At the time of our review, HHS reported that 4 states were not pursuing 
SACWIS development and most of these states reported various reasons in 
our survey for not developing a system. In Hawaii, the child welfare 
agency chose not to pursue SACWIS because it already had a statewide 
system in place that it believed was adequately meeting its needs and 
which was collecting and reporting federal child welfare data. After an 
attempt to develop a system. North Carolina cancelled its efforts because 
it could not build consensus across its 100 counties on the design of a 
uniform system. On our site visit to North Carohna, child welfare officials 
reported that they are currently working on a statewide information 
system that will encompass a number of social services, such as food 
stamps and mental health services, but an HHS official reported that North 
Carolina is not seeking federal SACWIS funding to support the 
development of this system. Vermont officials reported that they did not 
pursue SACWIS because the legislature declined to provide the matching 
state funds. In retrospect, they beheve that the choice not to develop 
SACWIS was best for the state because they found the SACWIS 
requirements too restrictive to enable the state to design a system to meet 
its needs. Officials said that the state would not use a number of the 
required SACWIS components, such as developing all the required 
electronic links to other agencies’ systems, especially since the state has a 
small child welfare population. Another state^North Dakota did not 
report in our survey the reason for stopping SACWIS development; 



Page 15 



20 



GAO-03-809 Child Welfare 



however, HHS officials reported that the state had attempted to develop a 
SACWIS, but faced a variety of problems, such as receiving state funding. 



States Accrue Benefits While most state child welfare agency officials said they recognize the 

from Using SACWIS, but benefits the state will achieve by developing SACWIS, such as enhancing 
Several Issues Create ability to track the whereabouts of foster children, 31 state agencies 

Delays in Completing behind the time frames they set for completion, with 26 states 

States’ Svstpms reporting delays ranging from 2 months to 8 years. State officials reported 

- ^ m our survey and during site visits that SACWIS has contributed to more 

efficient and effective agency functioning, which can improve states’ 
capabilities to manage their child welfare cases, including keeping track of 
where the children are living and the services they are receiving. Child 
welfare officials in Colorado reported that automation has improved 
agency functioning by making child welfare case information available 
statewide, which is especially helpful when families move from one county 
to another. In Oklahoma, caseworkers and state officials noted that they 
believe their children are safer since the implementation of SACWIS 
simply because the information on the children is easily accessible to the 
caseworkers and their supervisors. According to survey results, automated 
systems provided easier access to data and allowed caseworkers to better 

monitor children in their care, which may contribute to additional child 

welfare and administrative benefits, such as decreased incidences of child 
abuse and neglect, shortened length of time to achieve adoption, 
timeliness of payments to foster families, and timeliness of payn^ents to 
foster facilities (see table 5). New Jersey reported in our survey that its 
goal in developing a SACWIS is to integrate the more than 40 stand-alone 
systems that currently capture information on the children served by their 
child welfare agency. By pulling all of these systems together into a 
uniform SACWIS, the state hopes to improve the recording of casework 
activities in a timely manner and to develop a tool to better tcirget 
resources and services. Effectively integrating these systems will require 
the state to use a disciplined IT management approach that includes 
(1) detailed analyses of users’ needs and requirements, (2) a clearly 
defined strategy for addressing information needs, and (3) sufficient 
technical expertise and resources to support the effort. 
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Child welfare benefits 


Number of states identifying SACWIS 
as extremely or very effective in 
achieving measure 


Decreased incidence of child abuse and 
neglect 


25 


Shortened length of time to achieve adoption 


24 


Decreased recurrence of child maltreatment 


23 


Shortened length of time to achieve 
reunification 


22 


Management benefits 


Timeliness of payment to foster families 


36 


Timeliness of payment to foster facilities 


34 


Overall case management for children and 
families 


33 


Timeliness of child abuse and neglect 
investigations 


33 



Source: GAO survey. 



Note; Based on responses from 46 states developing or operating SACWIS. States not included 
answered “moderately effective,” “somewhat effective,” “not at all effective, not a state goal, 

“system doesn’t support," “don’t know," or “no answer.” 

Despite the benefits that many states have accrued with SACWIS, 31 states 
reported in our survey that they have been delayed in system completion 
beyond their initial deadline and identified a number of challenges that 
have led to the delay (see table 6).'" Some of the common difficulties states 
reported in developing SACWIS included receiving state funding approval, 
reaching internal agreement on system development, and creating a 
system that reflects child welfare work processes and is user friendly (see 
table 7). 



"twelve of the 46 states reporting that they are developing or operating a SACWIS 
reported that they have not experienced delays in developing their systems. In response to 
the length of the delays reported by 26 states in our survey, ACF commented on a draft of 
this report that these states may be using different definitions in defining their delays. 
However ACF did not provide further information on how the delays represented m this 
report differ from its perception of states’ experiences. In our survey, we asked states to 
report on the delays that exceeded the time Une outlined in their initial APD. 
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State 


Length of delay In months* 


Alabama 




36 


Arkansas 




6 


California 




36 


Colorado 




26 


Connecticut 




96 


District of Columbia 




36 


Georgia 




25 


Idaho 




21 


Illinois 




79 


Indiana 




6 


Kansas 




72 


Louisiana 




12 


Maryland 




12 


Michigan 




26 


Minnesota 




12 


Mississippi 




12 


New Jersey 




42 


New Mexico 




3 


Ohio 




36 


Oregon 




70 


Rhode Island 




14 


South Carolina 




47 


Tennessee 




36 


Utah 




48 


Virginia 




2 


Washington 




36 


Source: GAO survey. 






Note: While 31 states reported in the survey that they have experienced a delay in SACWIS 
development, only 26 states reported the length of their delay. The survey was issued in October 
2002 and completed by states as late as December 2002. 




were asked to report the number of months the delays exceeded the time line outlined in their 
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Table 7: Common SACWIS Development Challenges 



SACWIS development challenges 


Number of states experiencing at 
least some challenge” 


Receiving state funding approval 


42 


Reaching internal agreement on system 
development 


41 


Creating a system that reflects work 
processes 


40 


Creating a user friendly system 


39 


Insufficient state funding allocation 


32 


Securing contractors with knowledge of child 
welfare 


32 



Source; GAO survey. 



Note: Based on responses from 46 states developing or operating a SACWIS. 

“States were asked the extent to which certain factors were a challenge In developing SACWIS using 
the following scale: very great, great, moderate, some, or no challenge. The number of states 
reported in our analysis of challenges represents the total number of states reporting any extent the 
factor posed a challenge. States not included answered “no challenge,” “have not yet reached this 
stage,” “don’t know,” or “no answer.” 



Forty-two states reported challenges receiving funding approval, and 
32 states reported that insufficient state funding allocations for SACWIS 
development were a challenge in developing a comprehensive system. For 
example, Vermont officials reported that the state legislature declined to 
provide the matching state funds needed to secure federal funding for 
SACWIS. As a result, the state could not pursue development. In addition 
to the development challenges reported in our survey, 2 of the 5 states we 
visited reported that insufficient funding affected ongoing SACWIS 
maintenance. In Colorado, state agencies have received a series of budget 
cuts, which child welfare officials report have impacted their ability to 
respond to child welfare caseworkers’ needs for system improvements. In 
Iowa, child welfare officials reported that the state legislature 
appropriated $17,000 for state fiscal year 2002 for all child welfare 
automated systems activity, which they noted was an insufficient amount 
to maintain and upgrade systems as well as to pay staff. They reported 
that, as a result, the child welfare agency provided the information 
systems department with approximately $1 million from other parts of the 
agency’s budget. 

Despite user involvement in system design, some states still faced 
challenges trying to reach internal agreement among agency officials and 
caseworkers on the design of a system, resulting in a delay in 
development. In New York— a state where the counties are responsible for 
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administering child welfare services — the development of SACWIS was 
stalled when significant frustration with the system’s design led 
Commissioners from five large counties and New York City to request that 
the state stop SACWIS development until a reassessment of the design and 
plans for the implementation of the system was completed. After a 
thorough evaluation of the project, the state made changes to the project 
plan and developed statewide work groups to ensure all counties were in 
agreement with the system design. In addition, they hired a contractor to 
monitor system development and ensure that all users’ requirements are 
seriously considered. 



Similarly , despite states heavy reliance on contractors, many reported that 
securing contractors with knowledge of child welfare practice was a 
challenge for timely SACWIS development. Contractors are hired by the 
state for their system development knowledge but often are unfamiliar 
with child welfare policies and practices, especially since they vary from 
state to state. Officials in Colorado, for example, said they encountered 
difficulties with their contractors because of high turnover among the 
contractor staff and their lack of knowledge of child welfare policies. A 
contractor who has worked with 7 states to develop their SACWIS 
reported that contractors are asked to learn the child welfare business 
practices of a state in a short amount of time and that states cannot devote 
many resources, such as caseworkers, to help in the design process 
because caseworkers need to devote their time to providing services to 
children and families. Therefore, contractors often have to acquire 
knowledge on their own. 



Many states reported that creating a system that reflects child welfare 
work processes and is user friendly was a challenge in developing 
SACWIS. These issues were also identified in the federal reviews of states’ 
SACWIS. For example, one state explained in the SACWIS review that it 
had designed a system to meet the caseworkers’ needs and reflect the 
nature of the child welfare work processes by developing a system that 
required events to be documented as they occurred. However, this design 
limited the SACWIS’s functionality because it did not allow the 
caseworkers to go back and enter information after an event happened. 
The state explained that caseworkers do not use the system in real time, 
but provide services to the children and families and then record the 
information in the system. The state had to redesign the system to correct 
for this design flaw. In addition, the 14 states reporting that they have 
adapted a system from another state have experienced some challenge 
modifying the systems to reflect their work processes. While HHS advises 
states to consider adapting another state’s system if it requires few 



Page 20 



25 



GAO-03-809 Child Welfare 



changes, states report that they and their contractors were not always 
aware of the changes that would need to be made to adapt a system. 
Although Colorado and New York decided to modify another state’s 
SACWIS instead of designing a new system, child welfare officials in these 
states reported that adapting a SACWIS from another state has created 
more problems than anticipated. Colorado and New York adapted systems 
from state-administered child welfare agencies, which required extensive 
modifications to meet their needs as county-administered states. For 
example, Colorado needed a system that supported its administrative 
structure and could handle a larger number of cases. 



HHS Provides Some 
Assistance to Help States 
Meet SACWIS 
Requirements 



HHS has assisted states in a variety of ways in developing and completing 
their SACWIS." As a part of its regulatory responsibilities, HHS must 
review, assess, and inspect the planning, design, development, installation, 
and operation of SACWIS. In addition to reviewing and monitoring states’ 
APDs, HHS conducts on-site SACWIS reviews to comply with these 
responsibilities. HHS officials told us that these reviews are a detailed and 
thorough assessment of state systems to ensure the systems’ compliance 
with SACWIS requirements. In addition, officials reported that they 
provide verbal technical assistance during the on-site review to help states 
that do not fuUy conform with the applicable regulations and policies. At 
the time of our review, HHS had conducted 26 SACWIS reviews— 5 of 
which were determined as meeting all the requirements and classified as 
complete. HHS officials told us that since states have the flexibility to 
build a SACWIS that meets their needs, a large portion of the formal 
reviews concentrate on ensuring that the systems conform to state 
business practices. For example, while SACWIS regulations require that a 
state report all AFCARS data from their SACWIS, one state HHS reviewed 
relied on a separate state system to report data on the children served by 
the juvenile justice agency who are eligible for IV-E foster care funds. The 
state proved it had developed an automated process to merge data from 
both systems to compile a single AFCARS report that included children 
captured in both their SACWIS and juvenile justice systems. Therefore, 



"With regard to the budget difficulties that states reported facing, since 1994 the federal 
government has made a commitment to help states develop and mamtain their SAOWIb by 
matching 75 percent of states’ development funds through 1997 and providing ^ ongomg 
match of 50 percent of state funding for the development and mainten^ce of their 
systems. However, since the states’ legislatures must make the initial commitment to fund 
SACWIS, the federal government cannot assist state child welfare agencies with this 
challenge. 
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nno recognized that this nitJi me siaie s needs and 

determined the SACWIS to be complete and meeting all requirements. 



Few systems have been determined complete after an on-site review 
because of unresolved issues, such as not being able to build links to other 
state information systems or not implementing certain eligibility 
determination functions. To help states address some of these 
development challenges, the SACWIS review team provides the state with 
recommendations for complying with SACWIS requirements. For example, 
HHS observed during a review in one state that the SACWIS was available ’ 
statewide, but information collected in one county was not available to 
caseworkers in other counties. The federal officials offered 
recommendations to the state to meet the SACWIS requirement that all 
mformation be available statewide. In addition, HHS officials reported that 
once the draft report with the results of the SACWIS review are 
completed, federal staff schedule a conference call with the state officials 
to walk through the system’s deficiencies and offer guidance on how the 
state can move forward. 



HHS facilitates the sharing of information between states developing 
SACWIS through an automated system users group that allows state and 
federal officials to exchange information, ideas, and concerns. According 
to some state and HHS officials, the trust level at these meetings is very 
high, which promotes open discussions and also creates an atmosphere 
for informal dialogue with HHS. The systems users group developed out of 
another active group — the child welfare users group — when HHS solicited 
state representatives to help HHS define a model child welfare information 
system, which was later used as the basis for the SACWIS functional 
requirements after the passage of the 1993 legislation authorizing 
enhanced federal funding. State officials in Iowa and New York reported 
that the systems users group continues to play an important role in 
providing a forum for the honest exchange of information on SACWIS 
development. For example, child welfare and technical officials in New 
York said that the systems users group has been very beneficial because 
they have learned from other states’ positive and negative experiences in 
developing SACWlS, as well as the experiences unique to states with 
county-administered agencies. In addition to the users group, HHS officials 
also sponsor a listserv — an electronic maiUng list — that allows state 
officials to exchange information, and a monthly conference call with state 
information technology directors. Iowa child welfare information 
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technology officials said that they find the monthly SACWIS telephone 
conference call helpful because project managers discuss issues such as 
nromising practices and new regulations.^^ 



Technical assistance for SACWIS development is also avafiable to states 
through the National Resource Center for Information Technology m Child 
Welfare (Resource Center). According to survey results, 9 states said they 
used the Resource Center for assistance in developing SACWIS and 
14 states reported using it for help with SACWIS maintenance and 
improvements. According to Resource Center officials, they assist states 
with SACWIS development by helping states understand the technolo^ 
that is available for use, providing information on the automation of child 
welfare work and converting data, and reviewing the APD documentation. 
For example, the Resource Center offered technical assistance to 
Pennsylvania to help the state decide if it should continue development of 
its current SACWIS, abandon the SACWIS project and allow the counties 
to operate individual systems, or design a different SACWIS. The Resource 
Center evaluated the current SACWIS to determine if it could capture 
information based on the SACWIS regulations and if it was user friendly 
for the caseworker. Following the Resource Center s analysis, 
Pennsylvania decided to discontinue the existing SACWIS and develop a 
new SACWIS. When the Resource Center opened in 1999 5 years after 

many states started developing SACWIS-staff were not very familiar with 
many of the efforts states made during development. In an attempt to 
remedy this lack of knowledge on states’ issues developing SACWIS, 
Resource Center staff participated in some of the on-site SACWIS reviews 
conducted by HHS. Both HHS and Resource Center officials believe this 
exposure to the SACWIS systems enhanced the availability of technical 
assistance resources and knowledge available to the states. 



commenting on a draft of this report. HHS indicated that a Web resource is avadab e to 
states interested in learning about other states’ efforts to develop human semces^hild 
welfare food stamps, Temporary Assistance to Needy Families, child care and child 
suDDort enforcement-information systems at http://www.acf.hhs.gov/nhsitrc. 
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Several Factors Affect 
the States’ Ability to 
Ensure Reliable Data 
on Children’s 
Experiences, and 
Some of HHS’s 
Oversight and 
Assistance Is 
Problematic 



Several factors affect states’ ability to collect and report reliable'** data on 
children served by state child welfare agencies, and some problems exist, - 
such as a lack of clear and documented guidance, with HHS’s oversight 
and technical assistance. Almost all of the states responding to our survey 
reported that insufficient caseworker training and inaccurate and 
incomplete data entry affect the quality of the data reported to HHS.''' In 
addition, 36 of the 50 states that responded to our survey reported that 

^ matching their state data element definitions 
to HHS s data categories, affected the quality of the data that they report to 
the federal government. For example. North Carolina officials told us that 
while state policy mandates that they count every location in which a child 
resides, including hospital stays, AFCARS regulations say that hospital 
stays and other short-term placements should not be included in the count 
of foster care placements. In cases where state policy differs from federal 
policy, state officials must carefully re-format their data in order to meet 
eder^ reporting requirements. Similarly, during assessments of 6 states’ 
comphance with AFCARS reporting standards, HHS found that these 
issues affect data reliability. Despite the assistance that HHS offers to 
states, such as testing state data quality and providing the results to states 
to aid them in resubmitting data, states report ongoing challenges 

receiving clear and documented guidance and accessing technical 
assistance. 



Data are reliable when they are complete and accurate. A subcategoiy of accuracy is 
consistency. Consistency refers to the need to obtain and use data thaiare deS weU- 
d^ined enough to yield similar results in similar analysis. See U.S. General Accounting 
D.cTs^rSo2> ofCompule7--Processed Data, GAO-02-15G (Washington, 

‘^States were asked the extent to which certain problems may decrease the oualitv of the 
data submitted to AFCARS and NCANDS using [he following^scaTe greS g S 
moderate, some, and no affect. ® ® ’ 

49 statoSS responses from 

asked to development, were 
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Insufficient Caseworker 
Training and Inaccurate 
and Incomplete Data Entry 
Are the Most Common 
Factors That Affect Data 
Reliability 



Almost every state responding to our survey and all the states we visited 
reported that insufficient training for caseworkers and inaccurate and 
incomplete data entry affect the quality of the data reported to AFCARS 
and NCANDS (see fig. 1). Although most states reported these as separate 
factors, HHS and the states we visited foimd that insufficient training and 
inaccurate and incomplete data entry are often linked. For example, in 
official reviews of states’ information systems capability to capture data 
and report them to AFCARS, HHS advised states to offer additional 
training to caseworkers on several AFCARS data elements, such as 
recording the reasons for a child leaving foster care, to improve the 
accuracy of the data submitted. Similarly, Oklahoma reported that the 
state found that caseworkers were misinterpreting reports of policy 
violations by foster parents and inaccurately recording them as abuse or 
neglect allegations. However, state officials told us that training is typically 
one of the first programs cut when states face tight budget restrictions. 

For example, Iowa officials told us that training has been significantly 
reduced in recent years because of budget cuts and new workers may wait 
2 to 3 months before being trained how to enter data appropriately into 
their SACWIS. 
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States reporting at least some affect in data quality 
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Source: GAO survey. 



Notes: Based on responses from 50 states. 



I«® '‘9“''® 3'’® ® “I® states teat reported the issue had a very great 

vI™ nroaf ■ 7°^®'’f® ®'’ tee quality of state data submitted to HHS. 

^ ® responses are represented in the top section of each bar. Moderate and 

anTvLr!^ ®®®*'®" States not included 

answered no affect, don t know, or no answer." 



Inaccurate and incomplete data entry can also result from a number of 
other factors, such as caseworkers’ hesitation to ask famUies for sensitive 
information. For example, caseworkers in Oklahoma reported that they 
did not feet comfortable asking if a child’s mother was married at the time 
of birth or if a child is of Hispanic origin— both of which are required 
AFCARS data elements. In commenting on a draft of this report, Oklahoma 
added that caseworkers did not understand why the data elements were 
required and how the federal government used the information. In 
addition, Iowa state officials said that caseworkers may guess the racial 
backgrounds of children in their CtU"e or record them as unknown, 
especially when children come from mixed racial backgrounds, rather 
than asking the family for the information. HHS noted similar issues in 
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5 states that have had an AFCARS review.'® Caseworkers were 
inaccurately recording a child’s race as “unable to determine” even though 
this option should be selected only if the child s parents or relatives^^ 
cannot provide the information, such as when a child is abandoned.” 



Caseworkers, supervisors, and managers in the 5 states we visited 
reported that additional factors, such as difficulties balancing data entry 
with the time that they spend with the families and children, contributed 
to inaccurate or incomplete data entry. In addition, our recent work on 
caseworker recruitment cUid retention found that caseworkers struggle to 
balance the time they spend with children and data entry, and reportedly 
spend at least 50 percent of their time documenting case records.'* 
Supervisors in Iowa explained that since caseworkers are responsible for 
ensuring that children and their famiUes receive the services they need, the 
caseworkers tend to initially limit data entry to the information that is 
necessary to ensure timely payment to foster care providers, and complete 
all other data elements when the caseworkers have time. In addition, 
caseworkers in Colorado said that they are between 30 and 60 days behind 
in their data entry, so the information in the automated system may not 
accurately reflect the current circumstances of children in care. The 
caseworkers reported that they tend to concentrate only on entering data 
that will allow them to open a case in their SACWIS. HHS’s Inspector 
General recently issued a report in which more than two-thirds of the 
states reported that caseworkers’ workloads, turnover, a lack of training, 
Eind untimely and incomplete data entry affected the reporting of AFCARS 
data.'® 



"We reviewed AFCARS reports from 6 of the 8 states assessed by HHS— Arkansas, 
Connecticut, New Mexico, Texas, Vermont, and Wyoming. HHS conducted reviews m 
Delaware and West Virginia after we completed our analysis. 

”ln commenting on a draft of this report, ACF said that the finding from the AFCARS ^ 
reviews indicates that information is often defaulted to the response “unable to determme 
in order for the element not to fail the missing data standard, not that workers are 
recording “unknown”; however, the report findings we used in this analysis instruct states 
to fix the defaults and address caseworker practice by enhancing training on the correct 
use of “unable to determine” when noting a child’s race. 

'*See U.S. General Accounting Office, Child Welfare: HHS Could Play a Great^ Role m 
Helping Child Welfare Agencies Recruit and Retain Staff, GAO-03-357 (Washington, D.C.: 
Mar. 31, 2003). 

'"Department of Health and Human Services, Office of Inspector General, AdapHon and 
Foster Care Analysis and Reporting System (AFCARS): Challenges and Limitations, 
OEI-07-01-00660 (Washington, D.C.: Mar. 2003). 
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Technical Challenges, such 
as Matching State 
Definitions to Federal 
Definitions, Affect Data 
Reliability 



In addition to data quality being affected by caseworker issues, many 
states experienced technical challenges reporting their data to HHS. The 
problems reported by states are typically a result of challenges associated 
with data “mapping” — matching state data elements to the federal data 
elements. For example, 36 states reported in our survey that matching 
their state-defined data to HHS’s definitions affected the quality of the data 
reported to NCANDS and AFCARS. Similarly, 24 states reported that 
matching the more detailed data options available in their states’ 
information systems to the federal data elements affected the quality of 
the data reported to NCANDS. Twenty-nine states reported that this issue 
created challenges in reporting data to AFCARS. For example, foUowing 
an AFCARS assessment, HHS instructed a state that collects detailed 
information on children’s disabUities, such as Downs Syndrome, Attention 
Deficit Disorder, and eating disorders, to map the information to the more 
limited options in AFCARS, such as mental retardation and emotionally 
disturbed. The Inspector General’s report found that states faced similar 
challenges mapping their data to meet the AFCARS reporting 
requirements. 



In many cases, states have to balance state policy with federal 
requirements to ensure that they are reporting accurate data to AFCARS 
and NCANDS, but are not contradicting their state policies. For example, 
Texas officials reported that although the findings of their AFCARS review 
instructed them to modify their SACWIS to collect, map, and extract data 
on guardianship placements, the state does not support guardianship 
arrangements.^'’ In addition, a recent report from the Child Welfare League 
of America (CWLA) found that when reporting the number of times 
children move from one foster care placement to another, states varied in 
the type of placements included in that count.'* For example, 29 percent of 
the states responding to CWLA’s survey included respite," 25 percent 
included runaways, and 16 percent included trial home visits when 
reporting the number of placements a child had during the AFCARS report 
period. According to federal guidance, the “number of placements” 
element is meant to gather information on the number of times the child 



Guardianship arrangements occur when permanent legal custody of a child is awarded to 
an inclividual, such as a relative, but the child is not legally adopted. 

'‘Child Welfare League of America. National Working Group Highlights, “Placement 
onno^ Measure and Diverse Out-of-Home Care Populations” (Washington, D.C., Apr. 

22 

Respite care provides temporary chUdcare for children away from their caretakers. 
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welfare agency found it necessary to move a child while in foster care and 
that by including runaways or trial home visits, a state is infla,ting the 
number of moves a child experienced. However, North Carolina officials 
told us that although the federal definition for placements instructs states 
not to include such stays when counting the number of children’s foster 
care placements, the state instructs them to count each time a child is 
sleeping in a different place as a new placement. The Inspector General 
reported that the placement definitions were the most commonly cited 
source of confusion among the states surveyed.^ 

In addition to the challenges reported in our survey, HHS reported that 
transferring data from older data systems into SACWIS affects the quality 
of the data reported to AFCARS and NCANDS. HHS officials reported that 
they have observed that states experience the biggest change in data 
quality when they begin reporting from their SACWIS. In general, the first 
data submissions are of low quality because of the time it takes states to 
transfer data or the system re-sets the information for data elements. For 
example, in 1 state, 65 percent of the records reviewed by HHS during an 
AFCARS assessment recorded the date the children were removed from 
their homes as July 28, 1997— the date the SACWIS came on-line; however, 
the actual dates of removal for these children ranged from 1988 to 1997. 



Although HHS Has Taken 
Steps to Help States 
Improve Their Data, Some 
Problems with Its Efforts 
Exist 



HHS provides technical assistance for AFCARS and NCANDS reporting 
through a number of resources. HHS officials in the central office and 
NCANDS contractor staff serve as the points of contact for states to ask 
questions and seek guidance on reporting child welfare data. HHS officials 
reported that assistance is offered in a number of ways, including 
telephone and e-mail communication. The officials in 3 of the 5 states that 
we visited said that the one-on-one focused technical assistance was 
useful when provided in a timely fashion. Most state officials foimd the 
NCANDS data easier to report, in part because more people were available 
for consultation and they were more accessible and responsive. For 
example, states have access to four NCANDS specialists and staff in the 
contractor’s central office when they need assistance reporting child 
abuse and neglect information. However, some of the states we visited 



“Although the findings from the Inspector General’s report and our study are more recent, 
ACF commented on a draft of this report that it issued policy clanficaUons regarding 
placement information on July 5, 2002. 
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reported that only one or two staff in HHS’s central office are available to 
assist with AFCARS reporting. 



In addition, the Resource Center offers states assistance with improving 
data quality; however, Resource Center staff reported that the assistance is 
geared more towards improving the limited data used in the federal review 
process to monitor states’ compliance with child welfare laws and federal 
outcome measures — CFSR — rather than all the data reported to HHS. The 
Resource Center also sponsors an annual information technology 
conference during which sessions covering all data-related issues are held 
including practices for ensuring data quality and outcome evaluation in 
child welfare. 

In coiqunction with the national data conference, the HHS officials and the 
contractors that operate NCANDS hold an annual technical assistance 
meeting for states to share ideas with one another, discuss data elements 
that pose difficulties, and explore ways to address these problems. For 
example, at a recent technical assistance meeting, approximately 43 state 
representatives attended sessions on preparing the calendar year 2002 
NCANDS data submissions and received a detailed explanation of how the 
NCANDS staff test states’ data submissions for quality. In addition, an 
NCANDS state advisory group meets annually to talk with HHS officials 
about NCANDS data and their experiences reporting data. From these 
meetings, the state advisory group proposes changes or improvements to 
NCANDS. HHS and state officials reported that this partnership has helped 
ease some of the challenges in reporting child abuse and neglect data. 

In addition to the direct assistance through consultation with HHS officials 
and the Resource Center, HHS has made available to states the software it 
uses to examine states’ AFCARS and NCANDS submissions for 
inconsistencies and invaUd data. Officials in all the states we visited said 
that they regularly use this software, and an HHS official said that nearly 
every state has used the software at least once. When the data are 
submitted to HHS, they are run through the same software, and HHS 
notifies the states of areas where data are missing or inconsistent and 
allows the states to resubmit the data after errors are corrected. For 
example, HHS officials said that they worked with one state that was 
trying to determine the source of data errors in reporting to AFCARS the 
race or ethnicity of children in their care. The state was not able to 
detei^ne the source of the problem, so an HHS official examined the 
state s submissions and helped correct the data errors. The officials 
reported that these tests help them to identify some data quality errors, 
such as missing data, and said that they believe that, in general, data have 
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improved in recent years. However, they indicated that the tests cannot 
pinpoint the underlying problems contributing to these errors. 

Furthermore, one official reported that no specific efforts have been 
conducted to track the individual data; elements and, therefore, HHS 
cannot report on how data quality has changed over time. The results of 
these quality tests had been the basis for penalties levied against states 
that submitted low quality AFCARS data before the penalties were 
rescinded. HHS officials reported that the penalties served as an effective 
motivation to states to correct their data. Although HHS was not able to 
report how the lack of penalties might be affecting recent date quality, an 
official reported that the agency plans to conduct this analysis in the 

future. 

In an attempt to help states comply with the reporting standards and 
address some of the factors that contribute to date quality probleim, HHS 
performs comprehensive reviews of state information systems ability to 
capture AFCARS date to identify problems associated with data collection 
and reporting, and to ensure that the information in the automated system 
correctly reflects children’s experiences in care. The assessments include 
a technical review of the states’ computer code, a comparison of the date 
from selected cases available in the information system to the case files, 
and an improvement plan to resolve any errors. In addition, HHS officials 
offer guidance to the states on improvements that can be made to the 
information system and changes to program code used to report the 
AFCARS date. HHS conducted pilot reviews in eight states between 1996 
and 2000. By March 2003, HHS had conducted eight official reviews— even 
though states began reporting to AFCARS in 1995. According to results 
from six of the eight official AFCARS assessments we reviewed, no state 
met the reporting requirements for all AFCARS date elements. Table 8 
shows a selection of the data elements and the states’ ratings. 'The 
problems noted in the reviews are similar to those we heard from states 
responding to our survey and those we visited. For example, most states 
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Taw, e-. Selecl«l 



Data Elements 

Foster Ca re 

Child race 



Has the child been diagnosed with a disability? 
(yes, no, not yet determined) 



S"'»ti»na«y dlstufted, 

(does not apply, applies) 



Has child ever been adopted? 
(yes, no, unable to determine) 



Total number of removals from home' 



Date of discharge from previous episode' 



Date of latest removal' 



Date of placement in current setting 



Number of previo us placement settings in this episode' 

Current placement setting 

(pre-adoptive home, foster family home-relative, foster family home-nonrelative nmnn 
home, ins titution, supervised independent living, runawav. trial home ® ^ 



Out of State placement (yes, no) 



Rating factor* 
(# of states) 



2 (4 states) 

3 (1 state) 

4 (1 state) 



2 (6 states) 



2 (5 states) 
3 (1 state) 



1 (1 state) 

2 (3 states) 

3 (2 states) 



2 (3 states) 

3 (2 states) 
4 (1 state) 



2 (3 states) 

3 (2 states) 
4 (1 state) 



2 (3 states) 

3 (3 states) 

2 (5 states) 
4 (1 state) 

2 (6 states) 



2 (4 states) 

3 (1 state) 

4 (1 state) 



2 (4 states) 

3 (1 state) 

4 (1 state) 



retuSeSs^f(?AFr the state’s information system does not fully meet the 

rSu remenS for ^ 3, the information system 

\ “ tor ^CARS reporting are in place, but there are data entry nrobleins 

affecting the quality of data. According to an HHS official, data element^ that have a 

are rioted HHsiniThet'’ are rated as 2 until the technical issues 

A ‘ element as a 3 until the data entiy practices are 

1 if thP ^ receives a rating of 4 if in compUance with the AFCARS requirements or a 
1 If the requirement is not addressed in the state’s information system. ^ 



ERIC 



Page 32 



37 



GAO-03-809 Child Welfare 





Rating factor’ 


Data Elements 


(# of states) 


Most recent case plan goal 

(reunify with parents or principal care takers, live with other relative(s), adoption, long term 
foster care, emancipation, guardianship, case plan goal not yet established) 


1 (1 state) 
2 (5 states) 


Date of discharge^ 


2 (2 states) 

3 (2 states) 

4 (2 states) 


Reason for discharge' 

(not applicable, reunification with parent(s) or primary caretaker(s), living with other 
relative(s), adoption, emancipation, guardianship, transfer to another agency, runaway, 
death of child) 


2 (4 states) 

3 (1 state) 

4 (1 state) 


Adoption Data Elements 


Child race 

(American Indian or Alaska native, Asian, Black or African American, Native Hawaiian or 
other Pacific Islander, White, unable to determine) 


2 (4 states) 

3 (1 state) 

4 (1 state) 


Primary basis for determining special needs 

(not applicable; raclal/original background; age; membership in a sibling group; medical 
conditions or mental, physical, or emotional disabilities; other) 


2 (6 states) 


Mental retardation, physically disabled, emotionally disturbed" 
(does not apply, applies) 


2 (5 states) 
3 (1 state) 


Visually/hearing impaired 
(does not apply, applies) 


2 (4 states) 

3 (2 states) 


Other diagnosed condition 
(does not apply, applies) 


2 (6 states) 



Source: HHS. 



Note: Analysis based on results of AFCARS reviews in 6 states. States are rated on 66 foster care 
elements and 37 adoption elements. 

®HHS rates each data element using a four-point scale: (1) the AFCARS requirement(s) has not been 
implemented in the information system; (2) the technical system requirements for AFCARS reporting 
do not fully meet the standards; (3) the technical system requirements for AFCARS reporting are in 
place, but there are data entry problems affecting the quality of the data; (4) all of the AFCARS 
requirements have been met. 

‘’States are rated on each category. The data elements were combined for purposes of this analysis. 
'AFCARS data element used in the CFSR. 

State officials in these 6 states reported that they found the reviews useful 
for improving their AFCARS data submissions. In particular, they valued 
the thorough review by HHS officials of the computer code states use to 
report the data. Some of these officials reported that if all states were 
reviewed, the quality of data available in AFCARS would improve 
tremendously. However, HHS officials reported that they are not 
mandated to conduct the AFCARS reviews and that priority is placed on 
other reviews, such as the CFSR and SACWIS reviews. In addition, 
officials explained that the AFCARS reviews are not conducted in states 
developing SACWIS until the systems are operational. HHS expects to 
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complete approximately four reviews each year depending on available 
resources and has scheduled states through 2006. Similar to the SACWIS 
reviews, HHS officials offer recommendations and technical assistance to 
states during the review on how they can improve the quality of the data 
reported to AFCARS. 

Although the states we visited appreciated some of HHS’s efforts to assist 
with improving state data quality, they and most states responding to our 
survey agreed that the assistance is not always consistent or easily 
accessible (see fig. 2). States reported similar information to the Inspector 
General — AFCARS data elements were not clearly and consistently 
defined and technical assistance is effective but difficult to access. 



O 

ERIC 



Page 34 



39 



GAO-03-809 ChUd Welfare 



Figure 2: Federal Practices That Affect Data Quality 



States reporting at least some affect In data quality 



50 




Imprecise 
definitions for 
required fields 






AFCARS 

NCANDS 




Lack of clear, 
documented 
guidance 



35 




Difficulty 
accessing 
technical 
assistance 
and guidance 



Source: GAO survey. 



Notes: Based on responses from 50 states. 

The results reported in the figure are a sum of the states that reported the issue had a very great 
affect, great affect, moderate affect, or some affect on the quality of state data submitted to HHS. 
Very great and great affect responses are represented in the top section of each bar. Moderate and 
some affect responses are represented in the bottom section of each bar. States not included 
answered “no affect,” “don’t know,” or “no answer.” 



The primary concerns reported by the states we visited were delays in 
receiving clear, written guidance on defining and reporting certain data 
elements and the lack of state input in suggesting changes to AFCARS. 
Despite the written guidance available to states in the form of regulations 
and an on-line policy manual, states reported that the variation in state 
policies and practices make it difficult to inten^ret how to apply the 
general guidance. As a result, states consult with HHS to ensure they are 
applying the regulations appropriately. Hovyever, in commenting on a draft 
of this report, officials in Oklahoma told us that a common concern among 
the states is the lack of timely response from HHS when seeking guidance 
on how to report data. In addition, officials in New York explained they 
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have made it a practice to check the HHS Web site on a regular basis for 
current guidance, but have not found it a useful tool, and may turn to other 
states for guidance on AFCARS reporting. In commenting on a draft of this 
report, HHS explained that it first refers states to its Web site for 
information and believes that the avcdlable guidance addresses states’ 
concerns in most instances. In addition, the states that have had an 
AFCARS review experienced delays in obtaining guidance on how to 
proceed following the on-site review. Although they found the review to be 
very helpful, some states reported that HHS officials are delayed in 
responding to their questions. For example, Texas officials reported that 
the state sought clarification on its improvement plan and submitted 
additional questions to HHS following the review; however, when we 
spoke with the state officials, they said that they had been waiting 3 
months for a response on how to proceed. An HHS official told us that 
since the review process is relatively new, the agency is still developing a 
process to respond to the states and recognizes that it has not been 
responsive to the states already reviewed. In addition, HHS is taking steps 
to gather feedback from states and other users of AFCARS data to 
determine how to improve the system to make the data more accurate and 
useable. As a part of these efforts, HHS has published a Federal Register 
notice soliciting comments and held focus group meetings at national 
conferences. 

The difficulties states face in receiving federal guidance and assistance, as 
well as the other challenges they face in reporting data, may negatively 
impact the reliability of the data available in AFCARS and NCANDS. As a 
result, states are concerned that the national standards used in the CFSR 
are based on unreliable data and should not be used as a basis of 
comparison and potential financial penalty. The variation in states’ 
reporting practices may affect the validity of the measures and may place 
some states at a disadvantage. For example, the CWLA. and Inspector 
General studies found that approximately half the states include the 
juvenile justice population in their AFCARS reports, while the other states 
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do not."® Child welfare experts and some state officials believe that the 
states that include children served by the juvenile Justice agency in their 
AFCARS report may report a higher number of re-entries into the child 
welfare system or a higher number of moves within the system when 
compared to states that do not have IV-E agreements""’ with their juvenile 
justice systems."^ As a result, a state that includes such children in their 
AFCARS report are likely to fare less favorably when compared to the 
national standard than other states on two outcome measures— foster care 
re-entries and stability of foster care placements — and may face financial 
penalties associated with the CFSR. 



"^Federal guidance states that children who are (1) removed from their home and initially 
placed in a juvenile justice facility are not to be included in AFCARS reporting; (2) in a 
foster care setting who are moved to a juvenile justice facility and who are expected to be 
returned to a foster care setting should continue to be included in the AFCARS reporting 
population; and (3) in a foster care setting and are moved to a juvenile justice facility and 
who become the responsibility of another agency should not be included in the AFCARS 
reporting population. However, these studies show that the arrangements between child 
welfare and juvenile justice agencies vary, which affect the population of children repoited 
to AFCARS. For example, some state child welfare agencies have responsibility for all 
children in the juvenile justice system and include these children in their AFCARS reports, 
while other states only report children who are in the custody of the juvenile justice 
system, but receiving Title IV-E fimding. 

"^According to HHS, state child welfare agencies can claim Title IV-E foster care 
maintenance funds for eligible children for costs incurred by juvenile justice agencies that 
have entered into an agreement with the child welfare agencies. Delinquent children sei’ved 
by these agencies who meet the Title IV-E eligibility criteria are eligible for Title IV-E foster 
care maintenance funds. States must meet all Titles IV-B and IV-E program and/or eligibility 
requirements with respect to the children who are adjudicated delinquent and are receiving 
Title IV-E funding or for children in foster care under the placement and care responsibility 
of the child welfare agency. 

^^Children in the juvenile justice system may enter care because of behavior problems and 
return home when the behavior is controlled. However, the problem may arise again and 
the child could re-enter state custody. Also, the children may experience a greater number 
of placements while in foster care because of their behavior problems or a “step-down 
approach that some states use to graduaUy decrease the level of security required to care 
for the children. 
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States Are Using 
Various Practices to 
Overcome System 
Development 
Challenges and 
Improve Data on 
Children’s 
Experiences 



Some states are using a variety of practices to address the challenges 
associated with developing SACWIS and improving data reliability, 
although no formal evaluations are available on their effectiveness' To 
address the challenge of developing a system to meet statewide needs, 
states relied on caseworkers and supervisors from local offices to assist in 
the design and testing of the system. Few states reported in our survey 
strategies to overcome the other key challenges, such as limited funding 
and securing knowledgeable contractors, but some states we visited have 
devised some useful approaches. For example, Oklahoma child welfare 
officials — in order to maximize the limited state funding for maintaining 
their SACWIS — reported saving $1 million each year by hiring some of the 
contractors who developed their SACWIS as permanent staff. To improve 
data reliability, the 5 states we visited routinely review their data to 
identify data entry errors so that managers can ensure that the missing 
data are entered appropriately. In addition, some states reported that 
frequent use of the data, such as publishing periodic management reports 
detaihng local offices’ performance on outcome measures, helps 
caseworkers understand the importance of entering timely information. 



States Are Primarily 
Relying on SACWIS Users 
to Overcome Some of the 
Challenges to Completing 
Their Systems 



To overcome development challenges, survey respondents emphasized the 
importance of including system users in the various phases of completing 
SACWIS— planning, design, development, testing, and implementation. 
Past GAO work and other research efforts have determined similar 
approaches as best practices in building information systems.^ Forty-four 
of the 46 states responding to our survey that they are developing or 
operating a SACWIS indicated that they relied on internal users, such as 
caseworkers and supervisors, in the development of their systems and 34 
of these states said that they were extremely helpful participants. The 
extent to which the users were involved in development differed across 
the states. For example, in Texas, caseworkers from all of their child 
welfare regions were recmited to provide input on design and 
development, as well as during initial testing, pilot testing, and 
implementation of the system. Arkansas reported establishing a committee 
made up of users to review the work plan and sign off on recommended 
changes. In addition, states reported that their system users served a 



See U.S. Gener^ Accounting Office, Executive Guide: Improving Mission Performance 
Through Strategic Information Management and Technology, GAO/AIMD-94-115 
(Washington, D.C.: May 1, 1994); Center for Technology in Government, University of 
3ny, SUNY. Tying a Sensible Knot: A Practical Guide to State-Local Information 
Systems. Albany, N.Y., June 1997. 
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number of purposes, including serving as experts on the different 
specialties within child welfare, such as child abuse, foster care, or 
adoption, and as representatives from local or county offices to assist in 
identifying the diverse approaches to capturing information across the 
state. For example, Indiana reported that caseworkers involved in SACWIS 
development represented the unique needs of the different geographical 
areas of the state and helped design a uniform statewide system to meet 
the diverse needs of large, intermediate, and small local offices. 

Ten states noted that user input should not be limited to frontline workers, 
such as caseworkers, but should include representatives from other areas 
of the agency, such as the financial staff, and other agencies that serve 
children, such as child support enforcement.^*’ Since many SACWK link 
with other state information systems, states advised that developing a 
collaborative relationship with other state agencies will help the 
development of the system. While not one of the most conuuon challenges 
reported in our survey. New Hampshire reported that one of its challenges 
with meeting its SACWIS timeframe was not working collaboratively with 
other agencies, such as Temporary Assistance for Needy Families 
(TANF)’” and child support enforcement, to develop the payment 
component of SACWIS. SimilcU-ly, we previously reported that the 
difficulty developing linkages between social services agencies limits the 
effectiveness of all the programs to serve families.*" To attempt to 
overcome this challenge, 26 of the 46 states responding to our survey that 
they are developing or operating a SACWIS indicated that they included 
external public agency users and 23 reported using representatives from 
other state agencies that serve children in developing their SACWIS. 
Indiana said that a task force made up of representatives from the TANF 
and child support enforcement agencies was developed to design the 



^®The Child Support Enforcement Program is a joint federal, state, and local partnership 
that was established in 1975 under Title IV-D of the Social Security Act. Each state runs a 
child support program, which provides four major services: locating non-custodial parents, 
establishing paternity, establishing child support obligations, and collecting child support 
for families. 

^ 1996, the Congress created the block grant Temporary Assistance for Needy Families 
program replacing the Aid to Families with Dependent Children (AFDC) and related 
welfare programs. States were given increased flexibility in designing the eligibility criteria 
and benefit rules, which require work in exchange for time-limited benefits. 

^‘See U S General Accounting Office, Welfare Refoiin: Im.proving State Automated 
Systems Requires Coordinated Federal Effon, GAO/HEHS-00-48 (Washington, D.C.: Apr. 
27, 2000). 
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linkages between the systems. In addition, Colorado officials reported that 
they are working with the Department of Youth Corrections— an agency 
that shares the SACWIS with child welfare — to ensure that the shared 
screens use the same definitions. 

In addition to seeking input from caseworkers and other system users 
while developing SACWIS, many states continue to include users as a part 
of the unplementation teams, to serve as contacts in the field and provide 
ongoing assistance, and to provide input on system enhancements. 
Alabama responded in our survey that the state had “mentors" in each 
county to help caseworkers ac^ust to the new system. These mentors 
continue to provide ongoing support now that the system is implemented. 
Similarly, Oklahoma developed Field Implementation Teams consisting of 
one contractor and one child welfare staff person. During system 
implementation, the teams went to field offices to provide on-site 
assistance with using SACWIS and becoming accustomed with the new 
method of recording child welfare information. Fiirthermore, Oklahoma 
recruits experienced child welfare field staff for its SACWIS help desk 
because of their knowledge of the system and child welfare policy and 
practice. 

Although states faced other challenges in completing their SACWIS, few 
reported implementing approaches to overcome the barriers. According to 
survey results, a common problem states faced in developing SACWIS was 
receiving insufficient state funding for development. However, in our 
previous work on managing information technology, we found that the IT 
products can become obsolete in a matter of months rather than years, 
calling for more fi-equent investments in upgrades and enhancements.®" In 
addition, officials in Iowa told us that maintaining systems takes just as 
much money as building them. States did not report in our survey 
approaches for obtaining more funding for developing SACWIS, and few 
states reported developing strategies in an attempt to overcome the 
challenges associated with tight budgets for maintaining their systems. 

For example, Iowa officials engaged in careful planning with system users 
to ensure that they addressed the highest priorities when enhancing the 
system. In particular, the officials reported that maintaining tight control 
over the development and maintenance processes helps them avoid 
investing inordinate amounts of resources to make corrections to the 



®"See U.S. General Accounting Office, Managing Technology: Best Practices Can Improve 
Performance and Produce Results, GAO/T-AIMD-97-38 (Washington, D.C.: Jan. 31, 1997). 
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system. In Oklahoma, child welfare officials reported that they relied on 
the contractors who developed their SACWIS to conduct ongoing 
maintenance activities until the contract expired in 2001. At that time, the 
agency hired some of the contract staff as full-time state employees to 
continue with the maintenance activities. State officials explained that this 
approach ensured continuity of service, in addition to saving the agency 
approximately $1 million each year. Similarly, few states reported on 
approaches to overcome the challenge of finding contractors with 
knowledge of child welfare practice. However, Iowa officials explained 
that once the contract staff are hired, they are required to attend the same 
training as new caseworkers to ensure that they are familiar with the 
state’s child welfare policies and to familiarize themselves with casework 
practices. 



States Use Strategies, such 
as Producing Reports That 
Identify Missing Data, in an 
Attempt to Improve the 
Reliability of the Data 
Reported to HHS 



Twenty-eight states reported using approaches to help caseworkers 
identify the data elements that are required for federal reporting and to 
help them better understand the importance of entering timely and 
accurate data. Ten states responding to our survey reported reviewing the 
federal reporting requirements in training sessions as a promising 
approach they use to improve data quality or as a lesson learned. For 
example, Tennessee reported that the state added a component about 
AFCARS to the initial and ongoing training workers receive about using 
SACWIS. The curriculum addresses the AFCARS report in general and the 
individual data elements to help the caseworkers better understand the 
purpose of collecting the information. In Nebraska, a “desk aid” that 
explains the data elements and where and why to enter them in the system 
is available on the caseworkers’ computer desktops. In addition, New York 
has developed a step-by-step guide explaining to workers how NCANDS 
data should be entered, with references to the policy or statute requiring 
the information. 



To improve data reliability, some states have designed their information 
systems with special features to encourage caseworkers to enter the 
information. Four states responding to our survey and 3 states we visited 
designed their SACWIS with color-coded fields to draw attention to the 
data elements that caseworker are required to enter. For example, the 
AFCARS data fields in Oklahoma’s system are coded red until the data are 
entered, after which the fields change to blue. In addition, workers can 
look at a single screen in the Oklahoma system to see what AFCARS data 
elements need to be completed without having to scroll through the entire 
case record. Colorado, Iowa, New York, and Oklahoma have built into 
their systems alerts — also known as “ticklers”— to remind caseworkers 
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and supervisors of tasks that they need to complete. For example, in Iowa, 
alerts are sent to supervisors if a caseworker fails to enter the data 
necessary to complete a pa 5 unent to a foster care provider. Whereas, in 
Oklahoma, a stophght icon on the caseworker’s computer desktop 
reminds the worker when tasks are due. A green hght indicates that 
nothing is due within 5 days; a yellow light means that something is due 
within 5 days; and a red hght means that something is overdue. 
Caseworkers and supervisors in the states we visited had mixed responses 
about the usefulness and effectiveness of the alerts. Some caseworkers 
found them to be a nuisance, while other caseworkers and supervisors 
found them to be useful tools in managing workloads and prioritizing daily 
tasks. 

Six states reported that the best way to improve data quality was to use 
the data in published reports and hold the caseworkers and supervisors 
accountable for the outcomes of the children in their care. In addition, 6 
states responding to our survey reported using the data available in their 
information systems to measure state outcomes similar to the CFSR. State 
officials reported that this approach is an effective way to get local offices 
invested in the quality of the data. For example. North Carolina pubhshes 
montiily reports for each county comparing their performance on state 
data indicators, such as the length of time children spend in care, to 
counties of sunilar size and the state as a whole. County officials reported 
that these reports encourage workers to improve the quality of the data 
collected and entered into the state system since their performance is 
being widely published and compared to other counties. 

In addition, all the states we visited reported that frequent review of their 
data, such as using software from HHS to test their AFCARS and NCANDS 
data to pin-point data entry errors prior to submitting them to HHS, has 
helped improve data quality. When the states identify poor data, they alert 
the caseworkers and supervisors of needed corrections and data entry 
improvements. For example, Colorado runs these reports about 4 to 5 
times a year, with one run occurring approximately 6 weeks before each 
AFCARS submission. When the data specialists find errors, they notify the 
caseworker to clean up the data. New York officials told us that they 
incorporate the results fi-om these tests in training if a consistent pattern 
of errors is identified. 


Conclusion 


While most states are developing statewide information systems, 
challenges with data rehability remain. Although SACWIS development is 
delayed in many states, state officials recognize the benefits of having a 
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uniform system that enhances the states’ ability to monitor the services 
provided and the outcomes for children in their care. Although states 
began reporting to NCANDS in 1990 and were mandated to begin reporting 
to AFCARS in 1995, most states continue to face challenges providing 
complete, accurate, and consistent data to HHS. In addition, the results of 
more recent HHS efforts, such as conducting AFCARS-related focus 
groups, are unknown. Reliable data are essential to the federal 
government’s development of policies that address the needs of the 
children served by state child welfare agencies and its ability to assist 
states in improving child welfare system deficiencies. Without well- 
documented, clearer guidance and the completion of more comprehensive 
reviews of states’ AFCARS reporting capabilities, states are limited in 
overcoming challenges that affect data reliability. Because these 
challenges still remain, HHS may be using some questionable data as the 
foundation for national reports and national standards for the CFSR and 
may not have a clear picture of how states meet the needs of children in 
their care. 



Recommendation to 
the Secretary Of 
Health and Human 
Services 



To improve the reliability of state-reported child welfare data, we are 
recommending that the Secretary of HHS consider, in addition to HHS’s 
recent efforts to improve AFCARS data, ways to enhance the guidance and 
assistance offered to states to help them overcome the key challenges in 
collecting and reporting child welfare data. These efforts could include a 
stronger emphasis placed on conducting AFCARS reviews and more 
timely follow-up to help states implement their improvement plans or 
identifying a useful method to provide clear and consistent guidance on 
AFCARS and NCANDS reporting. 



Agency Comments 



We obtained comments on a draft of this report from the Department of 
Health and Human Services’ Administration for Children and Families 
(ACF). These comments are reproduced in appendix 111. ACF also 
provided technical clarifications, which we incorporated when 
appropriate. 

ACF generally agreed with our findings and commented that the report 
provides a useful perspective of the problems states face in collecting data 
and of ACF’s effort to provide ongoing technical assistance to improve the 
quality of child welfare data. In response to our recommendation, ACF 
said that we categorized its efforts as “recent” and did not recognize the 
long-term efforts to provide AFCARS and NCANDS related guidance to the 
states. Although we did not discuss each effort in depth, we do mention 
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the agency s ongoing efforts in our report. However, we refer to the recent 
efforts in the recommendation in recognition of the agency’s current 
activities to formally obtain, document, and incorporate feedback from the 
states with regard to collecting and reporting adoption and foster care 
data. ACF also noted in its comments that the data definitions need to be 
updated and revised and said it is currently in the process of revising the 
AFCARS regulations to further standardize the information states are to 
report— which we acknowledge in our report. In addition to the steps HHS 
is taking to further improve the AFCARS data, our recommendation 
encourages HHS to consider ways to enhance the ongoing guidance and 
assistance offered to states to help them overcome the key challenges in 
collecting and reporting child welfare data. ACF requested specific 
recommendations on approaches to overcome the difficulty of collecting 
and merging information from multiple state and county programs into a 
single national database. While there may be additional methodologies 
that the agency could use to overcome such challenges, our 
recommendation focuses on improving the guidance already offered to the 
states as a step to helping them better comply with the reporting 
requirements. 



In addition, ACF added that although staff turnover in state child welfare 
agencies is a significant contributor to data quality issues, we did not focus 
on this as a significant factor. ACF also commented that it is firmly 
committed to continue to support the states and to provide technical 
assistance and other guidance as its resources will permit. However, 
because we recently issued a detailed report on a variety of caseworker 
issues,® we primarily focused in this report on the key data entry 
challenges caseworkers face and refer readers to our previous work for 
additional information on challenges related to caseworker recruitment 
and retention and their affect on child welfare agencies. In commenting on 
our previous work prior to its release, HHS indicated that it does not have 
the authority to require states to address factors that contribute to staff 
turnover, such as high caseloads and said that it has limited resources to 
assist the states in the area of staff recruitment and retention. 

ACF commented that it provided increased funding to the National 
Resource Centers in fiscal year 2003, which they believe will improve 



“See U.S^ener^ Accounting Office, Child Welfare: HHS Covld Play a Greater Role in 

ChiM Welfare Agencies Recruit and Retain Staff GAO-03-367. (Washington, D.C.: 
Mar. ol, 2003). 
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ACF’s ability to provide assistance to the states. After receiving the draft 
report for conunent, HHS separately provided information on an 
additional service the National Resource Center for Information 
Technology in Child Welfare provides to states. In an effort to assist states 
with improving the quality of their AFCARS data, the Resource Center will 
review states* programming code used for AFCARS data. As of June 2003, 
HHS reported that the Resource Center provided this assistance to 
Arkansas, Louisiana, Mississippi, North Carolina, Nevada, New Jersey, and 
Rhode Island, and 3 States— Maryland, Michigan, and Wisconsin— and the 
District of Columbia have requested the assistance. 

In response to our survey methodology, ACF requested that we explain 
why the territory of Puerto Rico was not included in the state survey. 
Tilthough Puerto Rico receives federal child welfare funds, we traditionally 
focus on the states and therefore do not include the U.S. territories, 
including American Samoa, the Commonwealth of the Northern Mariana 
Islands, Guam, Puerto Rico, and the Virgin Islands, in the scope of our 
reviews. 

Finally, in response to our discussion of the AFCARS review process, ACF 
provided a few clarifications. During the course of our review, an HHS 
official characterized the AFCARS review process as relatively new and 
explained that the agency is still developing a process to respond to the 
states following the completion of the on-site review. When responding to 
a draft of this report, ACF disagreed with this characterization. ACF 
commented that the review process has been in place since 1996, pointing 
to the pilot reviews as evidence that the agency has a defined process. 
However, when we requested AFCARS reports for review, HHS explained 
that the states undergoing pilot reviews would be re-reviewed and that the 
official process was formalized in 2001 with the release of an AFCARS 
review guide and the start of the official reviews. In addition, ACF 
commented that SACWIS reviews do not take priority over AFCARS 
reviews. However, officials had previously explained that although 
SACWIS and AFCARS reviews can happen at the same time, in practice, 
the. AFCARS reviews are scheduled to occur in the states that are 
developing SACWIS after they have participated in a SACWIS review. 
Furthermore, ACF explained that states do not develop their improvement 
plan following the conclusion of the AFCARS review. Instead, ACF 
officials draft the plan for the state. Although state representatives had 
described a challenge in receiving timely feedback on their improvement 
plan, we have changed the language in the report to reflect ACF’s 
comment. 
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We also provided a copy of our draft to child welfare officials in the 5 
states we visited — Colorado, Iowa, North Carohna, New York, and 
Oklahoma. Iowa and New York had technical clarifications, which we in 
incorporated when appropriate. Oklahoma provided additional 
information, which was incorporated. Colorado had no suggested 
corrections or edits. North Carohna did not provide any comments. 



As agreed with your offices, unless you publicly announce its contents 
earher, we plan no further distribution of this report until 30 days after its 
issue date. At that time, we wiU send copies of this report to the Secretary 
of Health and Human Services, state child welfare directors, and other 
interested parties. We wiU make copies available to others on request. In 
addition, the report Avill be available at no charge on GAO’s Web site at 
http://www.gao.gov. If you or your staff have any questions or wish to 
discuss this material further, please call me at (202) 512-8403 or Diana 
Pietro wiak at (202) 512-6239. Key contributors to this report are listed in 
appendix IV. 




Cornelia M. Ashby 
Director, Education, Workforce, 
and Income Secmity Issues 
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Appendix I: Scope and Methodology 



To determine the progress states have made in developing Statewide 
Automated Child Welfare Information Systems (SACWIS), we surveyed all 
50 states and the District of Columbia through a Web-based survey. We 
pretested the survey instrument in Maryland and the District of Columbia. 
We received responses from 49 states and the District of Columbia. The 
state of Nevada did not respond to the survey. We discarded a question 
that asked states to report the date their advance planning document 
(APD) was approved by the Department of Health and Human Services 
(HHS). Due to a technical error, the date was truncated and a valid answer 
was not stored in the responses. Of the 50 survey responses, 46 were from 
states that are pumuing SACWIS development. The 4 states not developing 
SACWIS were asked to skip sections of the survey that asked about 
SACWIS development, system modifications, and supported services and 
links. We did not independently verify the survey responses. In addition, 
we visited 5 states to obtain more detailed and qualitative information 
regarding states’ experiences developing SACWIS. We conducted site 
visits in Colorado, Iowa, New York, North Carolina, and Oklahoma. We 
selected these states to represent a range of SAC WIS development stages, 
sizes of foster care populations, and geographic locations. During our site 
visits, we interviewed state and local child welfare staff, state and local 
staff that regularly exchange information with the child welfare agency, 
and private contractors. We also spoke with HHS staff in the central and 
regional offices. National Resource Center officials, contractors involved 
in SACWIS development, and child welfare experts from the Child Welfare 
League of America and the American Public Human Services Association. 

To determine how states and HHS ensure reliable data exist on children 
served by child welfare agencies we surveyed states using the above- 
mentioned survey instrument. In addition, we interviewed state and HHS 
officials on their efforts to analyze and compile data and HHS’s role in 
providing technical assistance to states. We spoke with state officials 
during our site visits and HHS officials in the central and regional offices 
and attended the 6th National Child Welfare Data Conference. We 
obtained and reviewed available SACWIS and Adoption and Foster Care 
Analysis and Reporting System (AFCARS) reports. At the time of our 
review, HHS had conducted 26 SACWIS reviews. We obtained and 
reviewed 23 reporte. The remaining reports were not available for review 
because HHS has not yet completed the report or shared the results with 
the state. Most of the SACWIS reports were considered drafts since many 
states are in the process of resolving issues with completing their systems. 
We reviewed AFCARS assessment reports from 6 of the 8 states assessed 
by HHS— Arkansas, Connecticut, New Mexico, Texas, Vermont, and 
Wyoming. HHS conducted reviews in Delaware and West Virginia after we 
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Appendix I; Scope and Methodology 



completed our analysis. We did not review any of the eight pilot review 
reports since these were not final reports and HHS plans to conduct 
official reviews in these states. These AFCARS assessment reports were 
anal 5 ^ed to understand the breadth of on-site assistance HHS provides to 
states during the review and to identify common data collection and 
reporting difficulties among states. Finally, we talked with officials in 6 of 
the 8 states that had an AFCARS review about their experiences during the 
review and child welfare experts. 

To identify practices state and local child welfare agencies are using to 
help ensure the accuracy, timeliness, and completeness of child welfare 
data we interviewed state and local child welfare officials on our site visits 
and inquired about the practices they are employing. We also included 
questions on practices and lessons learned in our survey. In addition, we 
spoke with numerous child welfare experts, including individuals from the 
National Resource Center for Information Technology in Child Welfare, 
the Child Welfare League of America, and the American Public Human 
Services Association.' 



The Chdd Welfare League of America is an association of almost 1,200 pubUc and private 
nonprofit agencies that assist abused and neglected children and their families with a wide 
range of services. The American PubUc Human Services Association is an organization of 
mdividuals and agencies concerned with human services, including state and local human 
service agencies and individuals who work in or otherwise have an interest in human 
service programs. 
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Appendix II: State SACWIS Stages of 
Development 



SACWIS stages of development 


Completed 


Partially operational 




Arizona 


Alabama 




Massachusetts 


Florida 




Oklahoma 


Illinois 




Rhode Island 


Mississippi 




West Virginia 


New York 




Oregon 


Operational 


South Carolina 




Arkansas 


Tennessee 




California 


Wisconsin 




Colorado 


Connecticut 


Implementation 




Delaware 


Alaska 




District of Columbia 


Maryland 




Idaho 


Indiana 


Planning 




Iowa 


Georgia 




Kentucky 


Kansas 




Maine 


Louisiana 




Michigan 


Missouri 




Minnesota 


New Jersey 




Montana 


Ohio 




Nebraska 


Pennsylvania 




Nevada 


New Hampshire 


No SACWIS 




New Mexico 


Hawaii 




South Dakota 


North Carolina 




Texas 


North Dakota 




Utah 


Vermont 




Virginia 


Washington 


Wyoming 


Source: HHS. 






Note: Status is as of May 22, 2003. 
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Appendix III: Comments from the 
Department of Health and Human Services 





DEPARTMENT OF HEALTH & HUMAN SERVICES 



ADMINISTRATION FOR CHILDREN AND FAMILIES 
Office of the Assistant Secretary, Suite 600 
370 L’Enfant Promenade. S.W. 

Washington, D.C, 20447 



DATE: “ 2 20(B 



TO: 



FROM: 



Cornelia M. Ashby 
Director, Education, Workforce, 
and Income Security Issues 



Wade F, Horn, 

Assistant Secretary 

for Children and Families 



SUBJECT: Comments on the GAO Draft Report: *‘Most States are Developing 

Statewide Information Systems, but the Reliability of Child Welfare Data 
Could be Enlianced" (GAO-03-809) 



Attached are the Administration for Children and Families’ comments on the GAO Draft 
Report: “Most States are Developing Statewide Information Systems, but the Reliability 
of Child Welfare Data could be Enhanced” (GAO-03-809). 

We appreciate the opportunity to comment on this report. Should you have any questions 
or need additional information, please contact Susan Orr, Associate Commissioner of the 
Children’s Bureau at (202) 205-8618. 
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COMMENTS OF THE ADMINISTR ATION FOR CHILDREN AND FAMILIES (ACF) 
ON THE GENERAL ACCOUNTING O FFICE’S (GAO) DRAFT REPORT: “MOST 
STATES ARE DEVELOPING STATEWIDE INFORMATION SYSTEMS. BUT THE 
RELIABILITY OF CHILD WELFARE DATA COULD BE ENHANCED .” GAO-03-809 

The Administration for Children and Families (ACF) appreciates the opportunity to comment on 
the GAO’s draft report. 

GAO Recommendation 

To improve the reliability of state-reported child welfare data, GAO recommends that the 
Secretary of HHS consider, in addition to HHS’s recent efforts to improve AFCARS data, ways 
to enhance the guidance and assistance offered to states to help them overcome the key 
challenges in collecting and reporting child welfare data. These efforts could include a stronger 
emphasis placed on conducting the Adoption and Foster Care Analysis and Reporting System 
(AFCARS) reviews and more timely follow-up to help states implement their improvement plans 
or identifying a useful method to provide clear and consistent guidance on AFCARS and the 
National Child Abuse and Neglect Data System (NCANDS) reporting. 

ACF Comments 

The background provided in the report was useful in giving a perspective of the problem states 
face in collecting data and ACF’s effort to provide ongoing technical assistance to improve the 
quality of that data. However, the GAO categorizes ACF’s efforts as “recent” and does not 
recognize the long-term efforts to provide AFCARS and NCANDS related guidance to the states. 
We recommend that the term “ongoing” be substituted for the term recent in the 
“Recommendation” section of the report. 

ACF has provided extensive guidance on how states can improve the quality of their AFCARS 
data since before the first AFCARS report was submitted. To categorize this effort as “recent” 
might be misconstrued by the reader. What follows is a short list of some of the guidance 
provided by this agency: 

• State Technical Advisory Group: 

March 1994 
May 1994 
March 1995 

• Statewide Automated Child Welfare Information Systems (SACWIS) Conferences: 
AFCARS sessions were held during these conferences to provide assistance and updates 
to states: 

September 1994 
September 1995 



iiSTCOPY AVAILABLE 



Page 61, 




GAO-03-809 Child Welfare 



Appendix III: Comments from the Department of Health and Human Services 



• ACF User Group Meetings: Held two times a year since 1994. AFCARS workshops 
were conducted as well as one-on-one meetings with states to provide technical 
assistance. 

• AFCARS Conferences/Meetings: 

State Technical Assistance Workshops: San Francisco and Washington D C. - July &. 
September 1994 

State Technical Assistance Workshops: San Francisco and Washington D.C. - July & 
September 1995 

SACWIS/AFCARS Conference - 1996 
AFCARS Data Conference -1997 
AFCARS Data Conference -1998 
AFCARS Data Conference -1999 
AFCARS Data Conference -2000 
AFCARS Data Conference -2001 
AFCARS Data Conference -2002 
AFCARS Data Conference -2003 

• Technical Assistance Documents: 

AFCARS: Part I - Strategic Planning Guide: A guide for state systems planning and 
development; Part II - Data Dictionary. 

AFCARS Guide to the Elements - September 1995 (Rescinded) 

NRC-ITCW AFCARS Toolkit (Available in draft form, final version to be released 
summer of 2003) 

Disability Code Table - Last updated 10/30/01 

State Data Compliance Software - First issued in 1995 

State Data Quality Software - First issued in 1996 

Statistical package of programming logic for the State Data Profile - 1998 

• Policy Guidance: 

Policy Interpretation Questions (PIQ) March 1994 
Policy Interpretation (^estions (PIQ) March 1995 

ACYF-PI-CB-95-09 (REVISED), issued 5/23/95: Methods of Extracting Data for 
Submission to AFCARS 

ACYF-CB-IM-00-04, issued March 30, 2000: Adoption and Foster Care Analysis 
and Reporting System Re-issuance of Technical Bulletins 
ACYF-CB-PI-01-07, issued 5/4/2001: Change in the State AFCARS File Name 
Child Welfare Policy Manual: September 24, 2001 (Replaces the two PIQs for 



AFCARS) 

ACYF-CB-IM-02-03, issued April 8, 2002: Decision regarding AFCARS Penalties, 
AFCARS File Name 

ACYF-CB-IM-02-05, issued April 22, 2002: AFCARS Assessment Reviews 
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ACF has issued several Technical Bulletins since 1994. These have been updated and modified 
as necessary. 



Old Technical Bulletins 


Revised Technical Bulletins 


Technical Bulletin U\: ACF BBS/AFCARS 
Support Menu (BBS=Bulletin Board Systems) 

(Deleted) 


Technical Bulletin U\: ACF Electronic 
Data Transmission 

(Previously Technical Bulletin U4) 


Technical Bulletin #2: File Format 


Technical Bulletin #2: AFCARS File Format 


Technical Bulletin #3: Optional Explanatory 
Footnotes 


Technical Bulletin #3: Optional Explanatory 
Footnotes and Questions to Guide Footnote 
Preparation 

(Previously Technical Bulletins #3 and #11) 


Technical Bulletin #4: Electronic Data Transmission 
(Revised Technical Bulletin #1) 


Technical Bulletin #4: Encryption Guidelines 
& Algorithms for State Use 

(Previously Technical Bulletins #6 and #9) 


Technical Bulletin #5; CONNECT: DIRECT 
Contacts 

(Deleted) 


Technical Bulletin #5: Record Identifier 
Encryption/Decryption Utility 

(Previously Technical Bulletin # 10) 


Technical Bulletin #6: Encryption Versus 
Sequencing 

(Corahined with Technical BnUetin U9 as Revised 
Technical Bulletin #4) 


Technical Bulletin #6: FCData Element 
Standards 

(Previously Technical Bulletin #8) 


Technical Bulletin #7: Adoption Data Element 
Standards 


Technical Bulletin #7: Adoption Data 
Element Standards 


Technical Bulletin #8: FC Data Element Standards 
(Revised Technical Bulletin #6) 


Technical Bulletin #8: State Data Compliance 
Utility 

(Previously Technical Bulletin #12) 


Technical Bulletin #9: Encryption Guidelines & 
Algorithms 

(Combined with Technical Bulletin #6 as Revised 
Technical Bulletin #4) 


Technical Bulletin #9: Data Quality Utility 
Version 1 .0 

(Previously Technical Bulletin #14) 


Technical Bulletin /^lO: Record Identifier 
Encryption/Decryption Utility: C Version (May 26, 
1995) COBOL Version 

(Revised Technical Bulletin /^5) 


Technical Bulletin #10: Receipt of AFCARS 
Report 

(New) 


Technical Bulletin #1 1 : Questions to Guide Footnote 
Preparation 

(Combined with Technical Bulletin #3 as Revised 
Technical Bnlletin #3) 


Technical Bulletin #11: Acceptance Test 
Guidance 

(Previously Technical Bulletin #13) 
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Technical Bulletin #12: State Penalty Calculation 
Utility (Revised Technical Bulletin #8) 




Technical Bulletin #13: Acceptance Test Guidance 
(Revised Technical Bulletin #11) 




Technical Bulletin #14: Data Quality Utility Version 
1.0 (Revised Technical Bulletin # 9) 





As noted in the GAO report, we concur with the general finding that states face significant 
challenges as they strive to improve the quality of the data that they report to the federal 
government. However, we are concerned that the report does not focus more attention on the 
continuing problem that the turnover of state and contractor staff has on the quality of data 
reported to the federal government. The GAO report briefly mentions the impact that the loss of 
social workers and contractor technical staff has on the quality of data, but the report does not 
focus on this as a significant factor. ACF has found that staff turnover to be a significant 
contributor to data quality issues. New social workers must be trained on how to effectively use 
the systems and what elements are important fi-om a data requirement perspective, while new 
technical staff must re-leam the intricacies of the national reporting requirements. ACF remains 
firmly committed to continue to support state technical staff and to provide technical assistance 
and other guidance as our resources will permit. We are continuously exploring new 
opportunities to extend our limited and shrinking resources to provide the types of technical 
assistance suggested in the GAO report to states and their staffs. 

Other Comments 

• Page 1, first paragraph, third sentence. 

“Through these federal matching funds, states can develop and implement Statewide 
Automated Child Welfare Information Systems (SACWIS) to manage their child welfare 
cases as well as to report child abuse and neglect, foster care, and adoption information to 
the federal government.” 

This sentence could be enhanced by noting that states have the option of building SACWIS or a 
non-SACWIS system to support their business processes. The GAO should consider replacing 
the current sentence with the following: 

“Through these federal matching funds, states can develop and implement information 
systems to manage their child welfare cases as well as to report child abuse and neglect, 
foster care, and adoption information to the federal government. If the state elects to 
pursue implementing a comprehensive case management system to support the different 
child welfare programs that are administered by the state, the state may elect to develop a 
system that meets the requirement for a Statewide Automated Child Welfare Information 
System (SACWIS). If a state elects to implement a SACWIS-compliant system, the state 
is eligible to receive additional funding to build and operate that system. Regardless of 
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the type of system that a state elects to implement, child welfare caseworkers at the 
county or local level are the key personnel who collect and document information on 
children and families served by child welfare agencies, in addition to performing a wide 
range of services to protect children, such as investigating child abuse and neglect reports 
or providing support services to maintain the children in their homes.** 

• Page 3, second paragraph, last sentence and page 4, first sentence. We are concerned about 
the example cited at the bottom of the page, which indicates that one state waited a year for 
technical assistance. 

“For example, state officials in Oklahoma said that it took HHS close to one year to 
provide written guidance on how to report the amount of monthly adoption assistance 
subsidies. As a result, the state may have misreported the amount of money the state 
awarded in subsidy payments to adoptive families.** 

We question whether the summary of the concern really conveys what happened. This is 
particularly true because issues or questions concerning the adoption assistance payment data 
elements are not generally complex or difficult to answer. 

• Page 12, first paragraph, second complete sentence. 

“As a result, Iowa designed a S ACWIS to reflect this work process by linking two 
databases - one to record child abuse and neglect information and one to record ongoing 
case records - that share information with one another.** 

We recommend that GAO modify this sentence to read as follows: 

“As a result, Iowa designed its SACWIS to have an interface with the state child abuse 
and neglect data system so that information can be shared between the two systems. 

Most states have built functionality that supports the investigation of child abuse and 
neglect allegations into their SACWIS.** 

ACF does not view the child abuse and neglect system in Iowa as part of its SACWIS. However, 
the state has met the SACWIS requirements in this area by building an interface between the two 
systems. 

• Page 12, second paragraph, first sentence. 

“Forty- three states reported in our survey that they have or are planning to incorporate 
child welfare and administrative components into their SACWIS (see Table 3).’* 

This sentence is somewhat confusing since many of the functional areas noted in the reference 
on page 13, Table 3, must be included in a SACWIS for it to be considered SACWIS- compliant. 
The issue is further complicated because some of the required functional areas identified in the 
table may be addressed through an interface to another system, while some must be included 
within the SACWIS application. The way this section is written, it may leave a reader with the 
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impression that some states are building SACWIS systems that do not meet all of the SACWIS 
requirements. We recommend that the first sentence in this paragraph be modified as follows: 

“Forty three states reported ... they are planning to incorporate optional child welfare and 
administrative components into their SACWIS (see Table 3).” 

This approach would probably necessitate the addition of a footnote explaining that some 
SACWIS components are mandatory while others are optional. Also, please see our comments 
below related to Table 3. 

• Page 12, second paragraph, second sentence. 

“Some of these components, such as child protection and determining IV-E eligibility, 
are required by SACWIS regulations, to the extent practicable.” 

The phrase “to the extent practicable” is used in the regulation and statute in relation to certain 
mandatory interfaces. The use of the phrase here is incorrect and it should be deleted. 

• Page 12, second paragraph, second sentence from bottom. 

“Currently, 27 states reported in our survey that they are at some stage of using their 
SACWIS to track independent living services, and an additional 14 states plan to include 
this component in their system in preparation for the legislative requirements.” 

The word "legislative" should be deleted. ACF is responsible for promulgating any National 
Youth in Transition Database (NYTD) requirements. 

• Page 13, Table 3. As previously noted, this table may leave the reader with the 
misunderstanding that some states are building non-compliant SACWIS systems. We 
provide the table below to identify our concerns. 

At the very least, we recommend that the GAO add the following disclaimer to this table: “HHS 
has indicated that a state’s failure to support most of these functional components would be a 
problem that the state would need to correct.” 



Service SACWIS Guidance 


Child Welfare Services 


Child protection 


Intake, screening, assessment, and investigation must be 
addressed in the SACWIS. Some sub^components 
related to these areas are optional. The components may 
be included in the SACWIS or addressed through the 
interface to a separate state child abuse and neglect 
system. Failure to support this functional area in the 
SACWIS application or through an interface would be a 
problem that the state would need to correct. 


Out of home care 


As defined in footnote “b,” out of home care is a 
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mandatory SACWIS component that must be supported 
in a SACWIS-compliant system. Some sub-components 
related to these areas are optional. Failure to support 
this functional area in the SACWIS application would 
be a problem that the state would need to correct. 


Adoption 


Adoption case management is a mandatory SACWIS 
component that must be supported in a SACWlS- 
compliant system. Some sub-components related to 
these areas are optional. Failure to support this 
functional area in the SACWIS application would be a 
oroblem that the state would need to correct. 


Independent living 


Independent living case management is a mandatory 
SACWIS component which must be supported in a 
SACWIS-compliant system. Some sub-components 
related to these areas are optional. Failure to support 
this functional area in the SACWIS application would 
be a problem that the state would need to correct. 


Intensive home based 
services 


Family support and family preservation related- 
activities are required to be supported in a SACWIS- 
compliant system. Failure to support these activities in 
the SACWIS application would be a problem that the 
state would need to correct. 


Administrative Services 


Workload 

management 


Workload management is a mandatory SACWIS 
component that must be supported in a SACWIS- 
compliant system. Some sub- components related to 
these areas are optional. Failure to support this 
functional area in the SACWIS application would be a 
problem that the state would need to correct. 


IV-E eligibility 


Title IV-E eligibility must be addressed in the SACWIS. 
Some sub-components related to this area are optional. 
The components may be included in the SACWIS or 
addressed through an interface to a separate state 
system. Failure to support this functional area in the 
SACWIS application or through an interface would be a 
problem that the state would need to correct. 


Foster care 
maintenance 
payments 
and 

Adoption assistance 
payments 


Both of these functional components must be addressed 
in the SACWIS. The components may be included in 
the SACWIS or addressed through an interface to a 
separate state fiscal system. Failure to support this 
functional area in the SACWIS application or through 
an interface would be a problem that the state would 
need to correct. 


Contract provider 
payment 


Contract management is an option that a state may elect 
to support through its SACWIS. 
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A payment to service providers is an option that the 
state may elect to support through its SACWTS or an 
interface to a state fiscal system. 



A payment to out of home care providers is a 
requirement that the state must support through its 
SACWIS or an interface to a state fiscal system. Failure 
to support this functional area in the SACWTS 
application or through an interface would be a problem 
that the state would need to correct. 



• Page 14, last sentence. This sentence states that North Dakota did not report in the GAO 
survey the reason for stopping SACWIS development. It is our understanding that North 
Dakota stopped its SACWIS initiative due to a combination of issues, including 
programmatic structure. 

• Page 17, Table 6, titled, “Number of Months States Delayed in SACWTS Development,” 
indicates states were asked to report the number of months the delays exceeded the time line 
outlined in their Advance Planning Document (ADP). Based on our experience with the 
states identified in Table 6, we believe that the respondents may have replied to the survey 
using different definitions of what “delayed” meant and different methodologies for counting 
the number of months that they were delayed. ACF recommends that GAO explore ways to 
enhance the reliability of the data presented in its report. 

• Page 23, footnote 13. This footnote states, “The analysis of survey responses about reporting 
data to HHS is based on responses from 49 states and the District of Columbia.” The GAO 
should consider explaining why Puerto Rico was excluded from the survey. 

• Page 24. The GAO should consider enhancing this section of the report to clarify the 
concerns being described. For example, the first two sentences state: 

“Almost every state responding to our survey and all the states we visited 
reported that insufficient training for caseworkers and inaccurate and incomplete data 
entry affect the quality of the data reported to AFCARS and NCANDS (see fig. 1.). 
Although most states did not report that insufficient training contributed to inaccurate and 
incomplete data entry, HHS and the states we visited found that these issues are often 
linked.” 

The first and second sentences appear to be contradictory. In sentence one the report indicates 
that the states reported that insufficient training for caseworkers affects the quality of the data, 
while the second sentence notes that most states did not report this to be the case. We 
recommend that the GAO enhance this area of its report to describe what was found. 

Furthermore, if this conclusion is primarily supported by empirical analysis, the GAO should 
consider downplaying the importance of this finding. 
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• Page 25, Figure 1 and Page 33, Figure 2. To clarify the survey results, we recommend that 
the bar chart include demarcation lines for states indicating the various factors had “some” or 
a “moderate” affect on the quality of data. The current presentation approach appears to 
present a more negative view of the impact these factors have on the quality of data than the 
survey questions may have solicited. 

• Page 26, first paragraph, first sentence. 

“Caseworkers were inaccurately recording a child’s race as unknown even though this 
option should be selected only if the child’s parents or relatives cannot provide the 
information, such as when a child is abandoned.” 

The finding from AFCARS reviews indicates that information is often defaulted to the response 

of “unable to determine” in order for the element not to fail the missing data standard, not that 

workers are recording “unknown.” 

• Pages 27 and 28. Having identified the difficulty of collecting information from multiple 
state and county programs into a single national database, ACF requests that GAO 
recommend solutions for this concern. Specifically, we request that the GAO describe 
methodologies on how to merge divergent data from multiple state and county programs with 
different program requirements and data definitions into a single common database in a 
manner that will result in better data. We welcome suggestions for an approach that would 
be more efficient, effective and economical than the approach currently used. 

• Pages 28 through 35. ACF has increased the accessibility of technical assistance resources 
available to the states through its support of the National Resource Centers (NRCs), 
specifically the National Resource Center for Information Technology in Child Welfare 
(NRC-ITCW). The Children’s Bureau (CB) increased funding to the NRC for FY 2003. The 
NRC- ITCW provides substantial technical assistance to states in data- related matters such as 
programming, input, interpretation, analysis, and reporting for AFCARS and NCANDS data. 
The increased funding will improve ACF’s ability to provide this assistance to our state 
partners. 

Concerning the states’ comment on the lack of state input in suggesting changes to AFCARS, 
it should be noted that AFCARS is defined in regulation. Therefore, it cannot be changed or 
modified without a process of public comment and a regulation change. 

All publicly available AFCARS documentation is accessible through ACF’s web page. In 
order to enable self-sufficiency and to sustain the state’s learning curve, state staff are asked 
to first read the guidance available through ACF’s web page. In most instances, the available 
guidance addresses their concern and we do not hear back from them. 

• Pages 32 and 33. ACF issued policy clarifications regarding placement information on 
July 5, 2002. Additionally, the AFCARS report is in the regulation process to further 
standardize the information that states are to report. We recognize that the definitions need 
to be updated and revised. 
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• Page 32, first paragraph, fourth sentence. The statement, “However, HHS officials reported 
that they are not mandated to conduct the AFCARS reviews and that priority is placed on 
other reviews, such as the CFSR and SACWIS review,” is incorrect. SACWIS reviews do 
not take precedence over AFCARS reviews. 

• Page 34. ACF wishes to advise the GAO that information pertaining to the AFCARS review 
process, specifically the improvement plan, is incorrect. States do not submit a proposed 
AFCARS improvement plan (AIP). The AIP is included in the final report to the state 
following an ^CARS assessment review (AAR). The AIP is based on the findings of the 
AAR and indicates how states are to proceed to correct the errors. States are given an 
opportunity to review the preliminary findings on-site and make corrections, negotiate 
changes, ^d ask for clarification at that time. The findings in the final report reflect the 
preliminary findings. In some instances, additional findings are made regarding the quality 
of the data after an analysis of the case file review is completed. States are to provide ACF 
with estimated due dates for the tasks listed in the AIP. 

Page 34, first p^graph, fifth sentence. This sentence, indicating that AFCARS reviews are 
relatively new, is incorrect. Reviews have been conducted either as pilots or actual reviews 
since 1996. The statement that the agency is still developing a process to respond to the 
states is also not accurate. There is a defined process for assessing state progress during the 
AFCARS improvement phase. This process was not part of the pilot program; therefore, it is 
being fine tuned and is a fluid process. Since AFCARS reviews are not mandatory, work 
related to the reviews on the federal level has to be adjusted to accommodate work on 
mandated priority items. 

Page 34, first paragraph, sixth sentence. This section discusses HHS taking steps to gather 
comments fix>m states and should be a separate paragraph. This is a separate topic than what 
is being discussed in the paragraph where it is located. 

• Page 35, footnote 22 has a few errors. 

1) The first sentence should be changed, as the juvenile justice agency does not claim 
federal title IV-E funds. The title IV-E agency claims costs incurred by the juvenile 
justice agency. 

2) The second sentence indicates that delinquents who meet IV-E eligibility criteria and 
"present with child protection and/or dependency issues, in addition to their delinquent 
status, may be eligible for title IV-E foster care.” This sentence should be revised to read 
that delinquents who meet title IV-E eligibility criteria are eligible for the program. 

3) The following clarification should be added at the end of the last sentence: "for children 
who are receiving title IV-E funding or for children in foster care under the placement 
and care responsibility of the child welfare agency.” As the sentence is written, it 
indicates diat all adjudicated delinquents must receive IV-E/IV-B protections, but that is 
only true if they are receiving IV-E fimds or meet our definition of foster care. 
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There are a couple of states that have an umbrella agency where the P/-B/IV-E agency 
has responsibility for all juvenile justice children. Therefore, although those children 
may not receive a foster care maintenance payment, the state may still have to meet the 
state plan requirements and protections for those youth if they are in foster care. 
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